Ch12_Optimization_Algorithms/Optimization_And_Deep_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization and Deep Learning\n",
    "\n",
    "In this section, we will discuss the relationship between optimization and deep learning as well as the challenges of using optimization in deep learning. For a deep learning problem, we will usually define a loss function first. Once we have the loss function, we can use an optimization algorithm in attempt to minimize the loss. In optimization, a loss function is often referred to as the objective function of the optimization problem. By tradition and convention most optimization algorithms are concerned with *minimization*. If we ever need to maximize an objective there's a simple solution - just flip the sign on the objective.\n",
    "\n",
    "## Optimization and Estimation\n",
    "\n",
    "Although optimization provides a way to minimize the loss function for deep\n",
    "learning, in essence, the goals of optimization and deep learning are\n",
    "fundamentally different. The former is primarily concerned with minimizing an\n",
    "objective whereas the latter is concerned with finding a suitable model, given a\n",
    "finite amount of data.  In Section 6.4,\n",
    "we discussed the difference between these two goals in detail. For instance,\n",
    "training error and generalization error generally differ: since the objective\n",
    "function of the optimization algorithm is usually a loss function based on the\n",
    "training data set, the goal of optimization is to reduce the training error.\n",
    "However, the goal of statistical inference (and thus of deep learning) is to\n",
    "reduce the generalization error.  To accomplish the latter we need to pay\n",
    "attention to overfitting in addition to using the optimization algorithm to\n",
    "reduce the training error. We begin by importing a few libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "%matplotlib inline\n",
    "import d2l\n",
    "from mpl_toolkits import mplot3d\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph below illustrates the issue in some more detail. Since we have only a finite amount of data the minimum of the training error may be at a different location than the minimum of the expected error (or of the test error)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 320.323437 184.15625\" width=\"320.323437pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 320.323437 184.15625 \n",
       "L 320.323437 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 58.523438 146.6 \n",
       "L 309.623438 146.6 \n",
       "L 309.623438 10.7 \n",
       "L 58.523438 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"maf8658c5a3\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"92.994925\" xlink:href=\"#maf8658c5a3\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0.6 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-46\"/>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "z\n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-54\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(85.043363 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"139.110628\" xlink:href=\"#maf8658c5a3\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 0.8 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "z\n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "z\n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "z\n",
       "\" id=\"DejaVuSans-56\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(131.159065 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-56\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"185.22633\" xlink:href=\"#maf8658c5a3\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 1.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(177.274768 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"231.342033\" xlink:href=\"#maf8658c5a3\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 1.2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(223.39047 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"277.457735\" xlink:href=\"#maf8658c5a3\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 1.4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-52\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(269.506173 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-120\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(181.114063 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-120\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"mc428accd9c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.523438\" xlink:href=\"#mc428accd9c\" y=\"144.697171\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1.25 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-8722\"/>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-53\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 148.49639)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"242.822266\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.523438\" xlink:href=\"#mc428accd9c\" y=\"119.133192\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- −1.00 -->\n",
       "      <g transform=\"translate(20.878125 122.93241)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"242.822266\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.523438\" xlink:href=\"#mc428accd9c\" y=\"93.569212\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −0.75 -->\n",
       "      <defs>\n",
       "       <path d=\"M 8.203125 72.90625 \n",
       "L 55.078125 72.90625 \n",
       "L 55.078125 68.703125 \n",
       "L 28.609375 0 \n",
       "L 18.3125 0 \n",
       "L 43.21875 64.59375 \n",
       "L 8.203125 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-55\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 97.368431)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-55\"/>\n",
       "       <use x=\"242.822266\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.523438\" xlink:href=\"#mc428accd9c\" y=\"68.005232\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- −0.50 -->\n",
       "      <g transform=\"translate(20.878125 71.804451)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"242.822266\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.523438\" xlink:href=\"#mc428accd9c\" y=\"42.441252\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- −0.25 -->\n",
       "      <g transform=\"translate(20.878125 46.240471)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"242.822266\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.523438\" xlink:href=\"#mc428accd9c\" y=\"16.877273\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.00 -->\n",
       "      <g transform=\"translate(29.257812 20.676491)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- risk -->\n",
       "     <defs>\n",
       "      <path d=\"M 41.109375 46.296875 \n",
       "Q 39.59375 47.171875 37.8125 47.578125 \n",
       "Q 36.03125 48 33.890625 48 \n",
       "Q 26.265625 48 22.1875 43.046875 \n",
       "Q 18.109375 38.09375 18.109375 28.8125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 20.953125 51.171875 25.484375 53.578125 \n",
       "Q 30.03125 56 36.53125 56 \n",
       "Q 37.453125 56 38.578125 55.875 \n",
       "Q 39.703125 55.765625 41.0625 55.515625 \n",
       "z\n",
       "\" id=\"DejaVuSans-114\"/>\n",
       "      <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-105\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-115\"/>\n",
       "      <path d=\"M 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 31.109375 \n",
       "L 44.921875 54.6875 \n",
       "L 56.390625 54.6875 \n",
       "L 27.390625 29.109375 \n",
       "L 57.625 0 \n",
       "L 45.90625 0 \n",
       "L 18.109375 26.703125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-107\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437 87.594531)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"41.113281\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "      <use x=\"68.896484\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"120.996094\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#pc2df9feb6c)\" d=\"M 69.937074 16.877273 \n",
       "L 72.242859 18.51536 \n",
       "L 74.548644 20.216038 \n",
       "L 76.854429 21.977533 \n",
       "L 79.160214 23.797948 \n",
       "L 81.465999 25.675265 \n",
       "L 83.771785 27.607348 \n",
       "L 86.07757 29.591942 \n",
       "L 88.383355 31.62668 \n",
       "L 90.68914 33.709083 \n",
       "L 92.994925 35.836563 \n",
       "L 95.30071 38.006427 \n",
       "L 97.606495 40.21588 \n",
       "L 99.91228 42.462028 \n",
       "L 102.218066 44.741882 \n",
       "L 104.523851 47.052363 \n",
       "L 106.829636 49.390301 \n",
       "L 109.135421 51.752446 \n",
       "L 111.441206 54.135466 \n",
       "L 113.746991 56.535956 \n",
       "L 116.052776 58.950437 \n",
       "L 118.358561 61.375368 \n",
       "L 120.664347 63.807143 \n",
       "L 122.970132 66.242101 \n",
       "L 125.275917 68.676528 \n",
       "L 127.581702 71.106663 \n",
       "L 129.887487 73.528704 \n",
       "L 132.193272 75.938811 \n",
       "L 134.499057 78.333114 \n",
       "L 136.804842 80.707714 \n",
       "L 139.110628 83.058694 \n",
       "L 141.416413 85.382119 \n",
       "L 143.722198 87.674046 \n",
       "L 146.027983 89.930525 \n",
       "L 148.333768 92.14761 \n",
       "L 150.639553 94.32136 \n",
       "L 152.945338 96.447845 \n",
       "L 155.251123 98.523156 \n",
       "L 157.556909 100.543404 \n",
       "L 159.862694 102.504731 \n",
       "L 162.168479 104.403315 \n",
       "L 164.474264 106.235372 \n",
       "L 166.780049 107.997165 \n",
       "L 169.085834 109.685009 \n",
       "L 171.391619 111.295277 \n",
       "L 173.697404 112.824403 \n",
       "L 176.00319 114.268889 \n",
       "L 178.308975 115.625313 \n",
       "L 180.61476 116.89033 \n",
       "L 182.920545 118.06068 \n",
       "L 185.22633 119.133192 \n",
       "L 187.532115 120.104789 \n",
       "L 189.8379 120.972496 \n",
       "L 192.143685 121.733439 \n",
       "L 194.449471 122.384857 \n",
       "L 196.755256 122.9241 \n",
       "L 199.061041 123.348639 \n",
       "L 201.366826 123.656067 \n",
       "L 203.672611 123.844103 \n",
       "L 205.978396 123.9106 \n",
       "L 208.284181 123.853546 \n",
       "L 210.589966 123.671069 \n",
       "L 212.895752 123.361439 \n",
       "L 215.201537 122.923075 \n",
       "L 217.507322 122.354543 \n",
       "L 219.813107 121.654567 \n",
       "L 222.118892 120.822025 \n",
       "L 224.424677 119.855954 \n",
       "L 226.730462 118.755556 \n",
       "L 229.036247 117.520195 \n",
       "L 231.342033 116.149404 \n",
       "L 233.647818 114.642885 \n",
       "L 235.953603 113.000511 \n",
       "L 238.259388 111.222328 \n",
       "L 240.565173 109.308555 \n",
       "L 242.870958 107.25959 \n",
       "L 245.176743 105.076004 \n",
       "L 247.482528 102.758549 \n",
       "L 249.788314 100.308154 \n",
       "L 252.094099 97.725925 \n",
       "L 254.399884 95.01315 \n",
       "L 256.705669 92.171294 \n",
       "L 259.011454 89.202002 \n",
       "L 261.317239 86.107095 \n",
       "L 263.623024 82.888573 \n",
       "L 265.928809 79.548614 \n",
       "L 268.234595 76.089568 \n",
       "L 270.54038 72.513963 \n",
       "L 272.846165 68.824495 \n",
       "L 275.15195 65.024033 \n",
       "L 277.457735 61.115616 \n",
       "L 279.76352 57.102446 \n",
       "L 282.069305 52.987891 \n",
       "L 284.37509 48.775478 \n",
       "L 286.680876 44.468895 \n",
       "L 288.986661 40.07198 \n",
       "L 291.292446 35.588728 \n",
       "L 293.598231 31.023277 \n",
       "L 295.904016 26.379912 \n",
       "L 298.209801 21.663057 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_13\">\n",
       "    <path clip-path=\"url(#pc2df9feb6c)\" d=\"M 69.937074 16.877273 \n",
       "L 72.242859 21.71463 \n",
       "L 74.548644 26.535801 \n",
       "L 76.854429 31.262176 \n",
       "L 79.160214 35.818852 \n",
       "L 81.465999 40.136436 \n",
       "L 83.771785 44.152703 \n",
       "L 86.07757 47.81408 \n",
       "L 88.383355 51.076912 \n",
       "L 90.68914 53.908479 \n",
       "L 92.994925 56.287747 \n",
       "L 95.30071 58.205822 \n",
       "L 97.606495 59.666111 \n",
       "L 99.91228 60.684166 \n",
       "L 102.218066 61.287238 \n",
       "L 104.523851 61.513534 \n",
       "L 106.829636 61.411206 \n",
       "L 109.135421 61.037089 \n",
       "L 111.441206 60.45523 \n",
       "L 113.746991 59.735226 \n",
       "L 116.052776 58.950437 \n",
       "L 118.358561 58.176098 \n",
       "L 120.664347 57.48738 \n",
       "L 122.970132 56.957458 \n",
       "L 125.275917 56.655624 \n",
       "L 127.581702 56.645492 \n",
       "L 129.887487 56.983349 \n",
       "L 132.193272 57.716673 \n",
       "L 134.499057 58.882882 \n",
       "L 136.804842 60.508318 \n",
       "L 139.110628 62.60751 \n",
       "L 141.416413 65.182723 \n",
       "L 143.722198 68.223814 \n",
       "L 146.027983 71.708387 \n",
       "L 148.333768 75.602255 \n",
       "L 150.639553 79.860189 \n",
       "L 152.945338 84.426941 \n",
       "L 155.251123 89.238513 \n",
       "L 157.556909 94.22364 \n",
       "L 159.862694 99.305461 \n",
       "L 162.168479 104.403315 \n",
       "L 164.474264 109.434642 \n",
       "L 166.780049 114.316928 \n",
       "L 169.085834 118.969653 \n",
       "L 171.391619 123.316181 \n",
       "L 173.697404 127.285573 \n",
       "L 176.00319 130.814244 \n",
       "L 178.308975 133.847451 \n",
       "L 180.61476 136.340562 \n",
       "L 182.920545 138.260076 \n",
       "L 185.22633 139.584375 \n",
       "L 187.532115 140.304185 \n",
       "L 189.8379 140.422727 \n",
       "L 192.143685 139.955577 \n",
       "L 194.449471 138.930212 \n",
       "L 196.755256 137.385271 \n",
       "L 199.061041 135.369543 \n",
       "L 201.366826 132.94071 \n",
       "L 203.672611 130.163866 \n",
       "L 205.978396 127.10987 \n",
       "L 208.284181 123.853546 \n",
       "L 210.589966 120.471799 \n",
       "L 212.895752 117.041676 \n",
       "L 215.201537 113.638432 \n",
       "L 217.507322 110.333639 \n",
       "L 219.813107 107.193396 \n",
       "L 222.118892 104.276669 \n",
       "L 224.424677 101.633816 \n",
       "L 226.730462 99.305324 \n",
       "L 229.036247 97.320799 \n",
       "L 231.342033 95.69822 \n",
       "L 233.647818 94.443489 \n",
       "L 235.953603 93.550279 \n",
       "L 238.259388 93.000189 \n",
       "L 240.565173 92.7632 \n",
       "L 242.870958 92.798419 \n",
       "L 245.176743 93.0551 \n",
       "L 247.482528 93.473906 \n",
       "L 249.788314 93.98839 \n",
       "L 252.094099 94.526655 \n",
       "L 254.399884 95.01315 \n",
       "L 256.705669 95.370564 \n",
       "L 259.011454 95.521765 \n",
       "L 261.317239 95.391738 \n",
       "L 263.623024 94.909478 \n",
       "L 265.928809 94.009785 \n",
       "L 268.234595 92.634924 \n",
       "L 270.54038 90.736101 \n",
       "L 272.846165 88.274726 \n",
       "L 275.15195 85.223429 \n",
       "L 277.457735 81.5668 \n",
       "L 279.76352 77.301842 \n",
       "L 282.069305 72.438123 \n",
       "L 284.37509 66.997617 \n",
       "L 286.680876 61.01425 \n",
       "L 288.986661 54.533151 \n",
       "L 291.292446 47.609632 \n",
       "L 293.598231 40.30792 \n",
       "L 295.904016 32.699675 \n",
       "L 298.209801 24.862327 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 58.523438 146.6 \n",
       "L 58.523438 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 309.623438 146.6 \n",
       "L 309.623438 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 58.523438 146.6 \n",
       "L 309.623438 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 58.523438 10.7 \n",
       "L 309.623438 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 141.305139 135.599571 \n",
       "Q 164.5777 138.000716 186.738131 140.287116 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "    <path d=\"M 182.964513 137.887156 \n",
       "L 186.738131 140.287116 \n",
       "L 182.553992 141.866035 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_14\">\n",
       "    <!-- empirical risk -->\n",
       "    <defs>\n",
       "     <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-101\"/>\n",
       "     <path d=\"M 52 44.1875 \n",
       "Q 55.375 50.25 60.0625 53.125 \n",
       "Q 64.75 56 71.09375 56 \n",
       "Q 79.640625 56 84.28125 50.015625 \n",
       "Q 88.921875 44.046875 88.921875 33.015625 \n",
       "L 88.921875 0 \n",
       "L 79.890625 0 \n",
       "L 79.890625 32.71875 \n",
       "Q 79.890625 40.578125 77.09375 44.375 \n",
       "Q 74.3125 48.1875 68.609375 48.1875 \n",
       "Q 61.625 48.1875 57.5625 43.546875 \n",
       "Q 53.515625 38.921875 53.515625 30.90625 \n",
       "L 53.515625 0 \n",
       "L 44.484375 0 \n",
       "L 44.484375 32.71875 \n",
       "Q 44.484375 40.625 41.703125 44.40625 \n",
       "Q 38.921875 48.1875 33.109375 48.1875 \n",
       "Q 26.21875 48.1875 22.15625 43.53125 \n",
       "Q 18.109375 38.875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.1875 51.21875 25.484375 53.609375 \n",
       "Q 29.78125 56 35.6875 56 \n",
       "Q 41.65625 56 45.828125 52.96875 \n",
       "Q 50 49.953125 52 44.1875 \n",
       "z\n",
       "\" id=\"DejaVuSans-109\"/>\n",
       "     <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-112\"/>\n",
       "     <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-99\"/>\n",
       "     <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "     <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-108\"/>\n",
       "     <path id=\"DejaVuSans-32\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(69.937074 134.471579)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"61.523438\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "     <use x=\"158.935547\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"222.412109\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"250.195312\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"291.308594\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"319.091797\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"374.072266\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"435.351562\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"463.134766\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"494.921875\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"536.035156\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"563.818359\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     <use x=\"615.917969\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 207.598032 74.086857 \n",
       "Q 207.927433 98.166852 208.241542 121.128918 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "    <path d=\"M 210.186642 117.101936 \n",
       "L 208.241542 121.128918 \n",
       "L 206.187016 117.156649 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_15\">\n",
       "    <!-- expected risk -->\n",
       "    <defs>\n",
       "     <path d=\"M 18.3125 70.21875 \n",
       "L 18.3125 54.6875 \n",
       "L 36.8125 54.6875 \n",
       "L 36.8125 47.703125 \n",
       "L 18.3125 47.703125 \n",
       "L 18.3125 18.015625 \n",
       "Q 18.3125 11.328125 20.140625 9.421875 \n",
       "Q 21.96875 7.515625 27.59375 7.515625 \n",
       "L 36.8125 7.515625 \n",
       "L 36.8125 0 \n",
       "L 27.59375 0 \n",
       "Q 17.1875 0 13.234375 3.875 \n",
       "Q 9.28125 7.765625 9.28125 18.015625 \n",
       "L 9.28125 47.703125 \n",
       "L 2.6875 47.703125 \n",
       "L 2.6875 54.6875 \n",
       "L 9.28125 54.6875 \n",
       "L 9.28125 70.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-116\"/>\n",
       "     <path d=\"M 45.40625 46.390625 \n",
       "L 45.40625 75.984375 \n",
       "L 54.390625 75.984375 \n",
       "L 54.390625 0 \n",
       "L 45.40625 0 \n",
       "L 45.40625 8.203125 \n",
       "Q 42.578125 3.328125 38.25 0.953125 \n",
       "Q 33.9375 -1.421875 27.875 -1.421875 \n",
       "Q 17.96875 -1.421875 11.734375 6.484375 \n",
       "Q 5.515625 14.40625 5.515625 27.296875 \n",
       "Q 5.515625 40.1875 11.734375 48.09375 \n",
       "Q 17.96875 56 27.875 56 \n",
       "Q 33.9375 56 38.25 53.625 \n",
       "Q 42.578125 51.265625 45.40625 46.390625 \n",
       "z\n",
       "M 14.796875 27.296875 \n",
       "Q 14.796875 17.390625 18.875 11.75 \n",
       "Q 22.953125 6.109375 30.078125 6.109375 \n",
       "Q 37.203125 6.109375 41.296875 11.75 \n",
       "Q 45.40625 17.390625 45.40625 27.296875 \n",
       "Q 45.40625 37.203125 41.296875 42.84375 \n",
       "Q 37.203125 48.484375 30.078125 48.484375 \n",
       "Q 22.953125 48.484375 18.875 42.84375 \n",
       "Q 14.796875 37.203125 14.796875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-100\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(173.697404 68.005232)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"61.507812\" xlink:href=\"#DejaVuSans-120\"/>\n",
       "     <use x=\"120.6875\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"184.164062\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"245.6875\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"300.667969\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "     <use x=\"339.876953\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"401.400391\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "     <use x=\"464.876953\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"496.664062\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"537.777344\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"565.560547\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     <use x=\"617.660156\" xlink:href=\"#DejaVuSans-107\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pc2df9feb6c\">\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"58.523438\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f(x): return x * np.cos(np.pi * x)\n",
    "def g(x): return f(x) + 0.2 * np.cos(5 * np.pi * x)\n",
    "d2l.set_figsize((4.5, 2.5))\n",
    "x = np.arange(0.5, 1.5, 0.01)\n",
    "fig ,= d2l.plt.plot(x, f(x))\n",
    "fig ,= d2l.plt.plot(x, g(x))\n",
    "fig.axes.annotate('empirical risk', xy=(1.02, -1.21), xytext=(0.5, -1.15),\n",
    "arrowprops=dict(arrowstyle='->'))\n",
    "fig.axes.annotate('expected risk', xy=(1.1, -1.05), xytext=(0.95, -0.5),\n",
    "arrowprops=dict(arrowstyle='->'))\n",
    "d2l.plt.xlabel('x')\n",
    "d2l.plt.ylabel('risk');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization Challenges in Deep Learning\n",
    "\n",
    "In this chapter, we are going to focus specifically on the performance of the\n",
    "optimization algorithm in minimizing the objective function, rather than a\n",
    "model's generalization error.  In :numref:`chapter_linear_regression`\n",
    "we distinguished between analytical solutions and numerical solutions in\n",
    "optimization problems. In deep learning, most objective functions are\n",
    "complicated and do not have analytical solutions. Instead, we must use numerical\n",
    "optimization algorithms. The optimization algorithms below all fall into this\n",
    "category.\n",
    "\n",
    "There are many challenges in deep learning optimization. Some of the most vexing ones are local minima, saddle points and vanishing gradients. Let's have a look at a few of them.\n",
    "\n",
    "### Local Minima\n",
    "\n",
    "For the objective function $f(x)$, if the value of $f(x)$ at $x$ is smaller than the values of $f(x)$ at any other points in the vicinity of $x$, then $f(x)$ could be a local minimum. If the value of $f(x)$ at $x$ is the minimum of the objective function over the entire domain, then $f(x)$ is the global minimum.\n",
    "\n",
    "For example, given the function\n",
    "\n",
    "$$f(x) = x \\cdot \\text{cos}(\\pi x) \\text{ for } -1.0 \\leq x \\leq 2.0,$$\n",
    "\n",
    "we can approximate the local minimum and global minimum of this function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 304.420313 184.15625\" width=\"304.420313pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 304.420313 184.15625 \n",
       "L 304.420313 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620313 146.6 \n",
       "L 293.720313 146.6 \n",
       "L 293.720313 10.7 \n",
       "L 42.620313 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m5b2f18399e\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"54.033949\" xlink:href=\"#m5b2f18399e\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −1.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-8722\"/>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-46\"/>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(41.892543 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"92.206646\" xlink:href=\"#m5b2f18399e\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −0.5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-53\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(80.065239 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.379343\" xlink:href=\"#m5b2f18399e\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(122.42778 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"168.552039\" xlink:href=\"#m5b2f18399e\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 0.5 -->\n",
       "      <g transform=\"translate(160.600477 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.724736\" xlink:href=\"#m5b2f18399e\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(198.773174 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"244.897433\" xlink:href=\"#m5b2f18399e\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(236.945871 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"283.07013\" xlink:href=\"#m5b2f18399e\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(275.118568 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_8\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-120\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(165.210938 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-120\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m7cb26f4eb1\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m7cb26f4eb1\" y=\"138.521358\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −1 -->\n",
       "      <g transform=\"translate(20.878125 142.320577)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m7cb26f4eb1\" y=\"97.824352\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257813 101.623571)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m7cb26f4eb1\" y=\"57.127346\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(29.257813 60.926565)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#m7cb26f4eb1\" y=\"16.43034\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(29.257813 20.229559)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-102\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"DejaVuSans-40\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "z\n",
       "\" id=\"DejaVuSans-41\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798438 87.271094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-102\"/>\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-40\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-120\"/>\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-41\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#p5a00c74f92)\" d=\"M 54.033949 57.127346 \n",
       "L 56.324311 58.523453 \n",
       "L 58.614672 60.246771 \n",
       "L 60.905034 62.260572 \n",
       "L 63.95885 65.329969 \n",
       "L 67.012666 68.749765 \n",
       "L 70.829935 73.365438 \n",
       "L 83.808652 89.415125 \n",
       "L 86.862468 92.76402 \n",
       "L 89.916284 95.794491 \n",
       "L 92.9701 98.450731 \n",
       "L 95.260461 100.170669 \n",
       "L 97.550823 101.641796 \n",
       "L 99.841185 102.854779 \n",
       "L 102.131547 103.804563 \n",
       "L 104.421909 104.490369 \n",
       "L 106.712271 104.915637 \n",
       "L 109.002632 105.087902 \n",
       "L 111.292994 105.018635 \n",
       "L 113.583356 104.72302 \n",
       "L 115.873718 104.219696 \n",
       "L 118.927534 103.263547 \n",
       "L 121.981349 102.036366 \n",
       "L 125.798619 100.222921 \n",
       "L 132.669704 96.608861 \n",
       "L 137.250428 94.307055 \n",
       "L 140.304244 92.96887 \n",
       "L 143.35806 91.869316 \n",
       "L 145.648421 91.239439 \n",
       "L 147.938783 90.803079 \n",
       "L 150.229145 90.581008 \n",
       "L 152.519507 90.590743 \n",
       "L 154.809869 90.846258 \n",
       "L 157.10023 91.357733 \n",
       "L 159.390592 92.131357 \n",
       "L 161.680954 93.16917 \n",
       "L 163.971316 94.468975 \n",
       "L 166.261678 96.024287 \n",
       "L 168.552039 97.824352 \n",
       "L 170.842401 99.854213 \n",
       "L 173.896217 102.884685 \n",
       "L 176.950033 106.23358 \n",
       "L 180.767302 110.764268 \n",
       "L 186.874934 118.440025 \n",
       "L 192.982565 126.00088 \n",
       "L 196.036381 129.492779 \n",
       "L 199.090197 132.65899 \n",
       "L 201.380559 134.76106 \n",
       "L 203.670921 136.585406 \n",
       "L 205.961282 138.094508 \n",
       "L 208.251644 139.253386 \n",
       "L 209.778552 139.815494 \n",
       "L 211.30546 140.199072 \n",
       "L 212.832368 140.396262 \n",
       "L 214.359276 140.40002 \n",
       "L 215.886184 140.204166 \n",
       "L 217.413091 139.80343 \n",
       "L 218.939999 139.1935 \n",
       "L 220.466907 138.371062 \n",
       "L 221.993815 137.333836 \n",
       "L 223.520723 136.080602 \n",
       "L 225.047631 134.611235 \n",
       "L 227.337993 132.004387 \n",
       "L 229.628354 128.921782 \n",
       "L 231.918716 125.377246 \n",
       "L 234.209078 121.390354 \n",
       "L 236.49944 116.986363 \n",
       "L 239.553256 110.519573 \n",
       "L 242.607071 103.454344 \n",
       "L 246.424341 93.940165 \n",
       "L 251.005065 81.833277 \n",
       "L 261.69342 53.205406 \n",
       "L 265.51069 43.791076 \n",
       "L 268.564505 36.90029 \n",
       "L 270.854867 32.204325 \n",
       "L 273.145229 27.980104 \n",
       "L 275.435591 24.284562 \n",
       "L 276.962499 22.140958 \n",
       "L 278.489406 20.270621 \n",
       "L 280.016314 18.6872 \n",
       "L 281.543222 17.403287 \n",
       "L 282.306676 16.877273 \n",
       "L 282.306676 16.877273 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620313 146.6 \n",
       "L 42.620313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 293.720313 146.6 \n",
       "L 293.720313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 293.720312 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620312 10.7 \n",
       "L 293.720312 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 108.214776 126.925918 \n",
       "Q 107.884314 118.462692 107.597474 111.116648 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "    <path d=\"M 105.755065 115.191636 \n",
       "L 107.597474 111.116648 \n",
       "L 109.752019 115.035568 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_14\">\n",
       "    <!-- local minimum -->\n",
       "    <defs>\n",
       "     <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-108\"/>\n",
       "     <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-111\"/>\n",
       "     <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-99\"/>\n",
       "     <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "     <path id=\"DejaVuSans-32\"/>\n",
       "     <path d=\"M 52 44.1875 \n",
       "Q 55.375 50.25 60.0625 53.125 \n",
       "Q 64.75 56 71.09375 56 \n",
       "Q 79.640625 56 84.28125 50.015625 \n",
       "Q 88.921875 44.046875 88.921875 33.015625 \n",
       "L 88.921875 0 \n",
       "L 79.890625 0 \n",
       "L 79.890625 32.71875 \n",
       "Q 79.890625 40.578125 77.09375 44.375 \n",
       "Q 74.3125 48.1875 68.609375 48.1875 \n",
       "Q 61.625 48.1875 57.5625 43.546875 \n",
       "Q 53.515625 38.921875 53.515625 30.90625 \n",
       "L 53.515625 0 \n",
       "L 44.484375 0 \n",
       "L 44.484375 32.71875 \n",
       "Q 44.484375 40.625 41.703125 44.40625 \n",
       "Q 38.921875 48.1875 33.109375 48.1875 \n",
       "Q 26.21875 48.1875 22.15625 43.53125 \n",
       "Q 18.109375 38.875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.1875 51.21875 25.484375 53.609375 \n",
       "Q 29.78125 56 35.6875 56 \n",
       "Q 41.65625 56 45.828125 52.96875 \n",
       "Q 50 49.953125 52 44.1875 \n",
       "z\n",
       "\" id=\"DejaVuSans-109\"/>\n",
       "     <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-105\"/>\n",
       "     <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-110\"/>\n",
       "     <path d=\"M 8.5 21.578125 \n",
       "L 8.5 54.6875 \n",
       "L 17.484375 54.6875 \n",
       "L 17.484375 21.921875 \n",
       "Q 17.484375 14.15625 20.5 10.265625 \n",
       "Q 23.53125 6.390625 29.59375 6.390625 \n",
       "Q 36.859375 6.390625 41.078125 11.03125 \n",
       "Q 45.3125 15.671875 45.3125 23.6875 \n",
       "L 45.3125 54.6875 \n",
       "L 54.296875 54.6875 \n",
       "L 54.296875 0 \n",
       "L 45.3125 0 \n",
       "L 45.3125 8.40625 \n",
       "Q 42.046875 3.421875 37.71875 1 \n",
       "Q 33.40625 -1.421875 27.6875 -1.421875 \n",
       "Q 18.265625 -1.421875 13.375 4.4375 \n",
       "Q 8.5 10.296875 8.5 21.578125 \n",
       "z\n",
       "M 31.109375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-117\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(71.593389 138.521358)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"88.964844\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"143.945312\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"205.224609\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"233.007812\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"264.794922\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "     <use x=\"362.207031\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"389.990234\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"453.369141\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"481.152344\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "     <use x=\"578.564453\" xlink:href=\"#DejaVuSans-117\"/>\n",
       "     <use x=\"641.943359\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 216.466383 71.343725 \n",
       "Q 215.445174 102.915153 214.46011 133.369131 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "    <path d=\"M 216.588381 129.43588 \n",
       "L 214.46011 133.369131 \n",
       "L 212.590472 129.306563 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_15\">\n",
       "    <!-- global minimum -->\n",
       "    <defs>\n",
       "     <path d=\"M 45.40625 27.984375 \n",
       "Q 45.40625 37.75 41.375 43.109375 \n",
       "Q 37.359375 48.484375 30.078125 48.484375 \n",
       "Q 22.859375 48.484375 18.828125 43.109375 \n",
       "Q 14.796875 37.75 14.796875 27.984375 \n",
       "Q 14.796875 18.265625 18.828125 12.890625 \n",
       "Q 22.859375 7.515625 30.078125 7.515625 \n",
       "Q 37.359375 7.515625 41.375 12.890625 \n",
       "Q 45.40625 18.265625 45.40625 27.984375 \n",
       "z\n",
       "M 54.390625 6.78125 \n",
       "Q 54.390625 -7.171875 48.1875 -13.984375 \n",
       "Q 42 -20.796875 29.203125 -20.796875 \n",
       "Q 24.46875 -20.796875 20.265625 -20.09375 \n",
       "Q 16.0625 -19.390625 12.109375 -17.921875 \n",
       "L 12.109375 -9.1875 \n",
       "Q 16.0625 -11.328125 19.921875 -12.34375 \n",
       "Q 23.78125 -13.375 27.78125 -13.375 \n",
       "Q 36.625 -13.375 41.015625 -8.765625 \n",
       "Q 45.40625 -4.15625 45.40625 5.171875 \n",
       "L 45.40625 9.625 \n",
       "Q 42.625 4.78125 38.28125 2.390625 \n",
       "Q 33.9375 0 27.875 0 \n",
       "Q 17.828125 0 11.671875 7.65625 \n",
       "Q 5.515625 15.328125 5.515625 27.984375 \n",
       "Q 5.515625 40.671875 11.671875 48.328125 \n",
       "Q 17.828125 56 27.875 56 \n",
       "Q 33.9375 56 38.28125 53.609375 \n",
       "Q 42.625 51.21875 45.40625 46.390625 \n",
       "L 45.40625 54.6875 \n",
       "L 54.390625 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-103\"/>\n",
       "     <path d=\"M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "M 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-98\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(176.186579 65.266748)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-103\"/>\n",
       "     <use x=\"63.476562\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"91.259766\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"152.441406\" xlink:href=\"#DejaVuSans-98\"/>\n",
       "     <use x=\"215.917969\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"277.197266\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"304.980469\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"336.767578\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "     <use x=\"434.179688\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"461.962891\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"525.341797\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"553.125\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "     <use x=\"650.537109\" xlink:href=\"#DejaVuSans-117\"/>\n",
       "     <use x=\"713.916016\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p5a00c74f92\">\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"42.620313\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-1.0, 2.0, 0.01)\n",
    "fig, = d2l.plt.plot(x, f(x))\n",
    "fig.axes.annotate('local minimum', xy=(-0.3, -0.25), xytext=(-0.77, -1.0),\n",
    "arrowprops=dict(arrowstyle='->'))\n",
    "fig.axes.annotate('global minimum', xy=(1.1, -0.95), xytext=(0.6, 0.8),\n",
    "arrowprops=dict(arrowstyle='->'))\n",
    "d2l.plt.xlabel('x')\n",
    "d2l.plt.ylabel('f(x)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective function of deep learning models usually has many local optima. When the numerical solution of an optimization problem is near the local optimum, the numerical solution obtained by the final iteration may only minimize the objective function locally, rather than globally, as the gradient of the objective function's solutions approaches or becomes zero. Only some degree of noise might knock the parameter out of the local minimum. In fact, this is one of the beneficial properties of stochastic gradient descent where the natural variation of gradients over minibatches is able to dislodge the parameters from local minima.\n",
    "\n",
    "### Saddle Points\n",
    "\n",
    "Besides local minima, saddle points are another reason for gradients to vanish. A [saddle point](https://en.wikipedia.org/wiki/Saddle_point) is any location where all gradients of a function vanish but which is neither a global nor a local minimum. Consider the function $f(x) = x^3$. Its first and second derivative vanish for $x=0$. Optimization might stall at the point, even though it is not a minimum.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 304.420313 184.15625\" width=\"304.420313pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 304.420313 184.15625 \n",
       "L 304.420313 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620313 146.6 \n",
       "L 293.720313 146.6 \n",
       "L 293.720313 10.7 \n",
       "L 42.620313 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"mbebd801061\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"54.033949\" xlink:href=\"#mbebd801061\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-8722\"/>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(46.662855 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"111.245159\" xlink:href=\"#mbebd801061\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −1 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(103.874065 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"168.456369\" xlink:href=\"#mbebd801061\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(165.275119 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"225.667578\" xlink:href=\"#mbebd801061\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(222.486328 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"282.878788\" xlink:href=\"#mbebd801061\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(279.697538 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-120\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(165.210938 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-120\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"mea8c43c5b5\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#mea8c43c5b5\" y=\"117.083786\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-53\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 120.883005)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#mea8c43c5b5\" y=\"78.185551\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257813 81.98477)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620313\" xlink:href=\"#mea8c43c5b5\" y=\"39.287316\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(29.257813 43.086535)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-102\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"DejaVuSans-40\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "z\n",
       "\" id=\"DejaVuSans-41\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798438 87.271094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-102\"/>\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-40\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-120\"/>\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-41\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_9\">\n",
       "    <path clip-path=\"url(#p753b4de644)\" d=\"M 54.033949 140.422727 \n",
       "L 58.038734 134.113877 \n",
       "L 62.043518 128.246459 \n",
       "L 66.048303 122.804464 \n",
       "L 70.053088 117.77188 \n",
       "L 74.057872 113.132698 \n",
       "L 78.062657 108.870906 \n",
       "L 82.067442 104.970495 \n",
       "L 86.072226 101.415453 \n",
       "L 90.077011 98.18977 \n",
       "L 94.081796 95.277436 \n",
       "L 98.08658 92.66244 \n",
       "L 102.091365 90.328771 \n",
       "L 106.09615 88.26042 \n",
       "L 110.100935 86.441375 \n",
       "L 114.105719 84.855626 \n",
       "L 118.110504 83.487163 \n",
       "L 122.687401 82.16873 \n",
       "L 127.264297 81.089289 \n",
       "L 131.841194 80.224939 \n",
       "L 136.990203 79.47989 \n",
       "L 142.711324 78.894472 \n",
       "L 149.004557 78.491322 \n",
       "L 156.442014 78.257598 \n",
       "L 166.740032 78.185761 \n",
       "L 182.187059 78.078005 \n",
       "L 189.624516 77.791489 \n",
       "L 195.917749 77.325184 \n",
       "L 201.63887 76.667649 \n",
       "L 206.787879 75.845721 \n",
       "L 211.364776 74.903513 \n",
       "L 215.941673 73.73725 \n",
       "L 220.51857 72.323035 \n",
       "L 224.523354 70.86341 \n",
       "L 228.528139 69.179637 \n",
       "L 232.532924 67.255707 \n",
       "L 236.537708 65.075609 \n",
       "L 240.542493 62.623332 \n",
       "L 244.547278 59.882866 \n",
       "L 248.552062 56.8382 \n",
       "L 252.556847 53.473324 \n",
       "L 256.561632 49.772226 \n",
       "L 260.566416 45.718898 \n",
       "L 264.571201 41.297328 \n",
       "L 268.575986 36.491505 \n",
       "L 272.58077 31.28542 \n",
       "L 276.585555 25.663061 \n",
       "L 280.59034 19.608419 \n",
       "L 282.306676 16.877273 \n",
       "L 282.306676 16.877273 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620313 146.6 \n",
       "L 42.620313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 293.720313 146.6 \n",
       "L 293.720313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 293.720312 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620312 10.7 \n",
       "L 293.720312 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 169.255408 105.480032 \n",
       "Q 168.886954 93.611456 168.553193 82.860375 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "    <path d=\"M 166.678274 86.920508 \n",
       "L 168.553193 82.860375 \n",
       "L 170.676348 86.79639 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_11\">\n",
       "    <!-- saddle point -->\n",
       "    <defs>\n",
       "     <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-115\"/>\n",
       "     <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "     <path d=\"M 45.40625 46.390625 \n",
       "L 45.40625 75.984375 \n",
       "L 54.390625 75.984375 \n",
       "L 54.390625 0 \n",
       "L 45.40625 0 \n",
       "L 45.40625 8.203125 \n",
       "Q 42.578125 3.328125 38.25 0.953125 \n",
       "Q 33.9375 -1.421875 27.875 -1.421875 \n",
       "Q 17.96875 -1.421875 11.734375 6.484375 \n",
       "Q 5.515625 14.40625 5.515625 27.296875 \n",
       "Q 5.515625 40.1875 11.734375 48.09375 \n",
       "Q 17.96875 56 27.875 56 \n",
       "Q 33.9375 56 38.25 53.625 \n",
       "Q 42.578125 51.265625 45.40625 46.390625 \n",
       "z\n",
       "M 14.796875 27.296875 \n",
       "Q 14.796875 17.390625 18.875 11.75 \n",
       "Q 22.953125 6.109375 30.078125 6.109375 \n",
       "Q 37.203125 6.109375 41.296875 11.75 \n",
       "Q 45.40625 17.390625 45.40625 27.296875 \n",
       "Q 45.40625 37.203125 41.296875 42.84375 \n",
       "Q 37.203125 48.484375 30.078125 48.484375 \n",
       "Q 22.953125 48.484375 18.875 42.84375 \n",
       "Q 14.796875 37.203125 14.796875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-100\"/>\n",
       "     <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-108\"/>\n",
       "     <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-101\"/>\n",
       "     <path id=\"DejaVuSans-32\"/>\n",
       "     <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-112\"/>\n",
       "     <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-111\"/>\n",
       "     <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-105\"/>\n",
       "     <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-110\"/>\n",
       "     <path d=\"M 18.3125 70.21875 \n",
       "L 18.3125 54.6875 \n",
       "L 36.8125 54.6875 \n",
       "L 36.8125 47.703125 \n",
       "L 18.3125 47.703125 \n",
       "L 18.3125 18.015625 \n",
       "Q 18.3125 11.328125 20.140625 9.421875 \n",
       "Q 21.96875 7.515625 27.59375 7.515625 \n",
       "L 36.8125 7.515625 \n",
       "L 36.8125 0 \n",
       "L 27.59375 0 \n",
       "Q 17.1875 0 13.234375 3.875 \n",
       "Q 9.28125 7.765625 9.28125 18.015625 \n",
       "L 9.28125 47.703125 \n",
       "L 2.6875 47.703125 \n",
       "L 2.6875 54.6875 \n",
       "L 9.28125 54.6875 \n",
       "L 9.28125 70.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-116\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(138.706539 117.083786)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-115\"/>\n",
       "     <use x=\"52.099609\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"113.378906\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "     <use x=\"176.855469\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "     <use x=\"240.332031\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"268.115234\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"329.638672\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"361.425781\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"424.902344\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"486.083984\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"513.867188\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"577.246094\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p753b4de644\">\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"42.620313\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-2.0, 2.0, 0.01)\n",
    "fig, = d2l.plt.plot(x, x**3)\n",
    "fig.axes.annotate('saddle point', xy=(0, -0.2), xytext=(-0.52, -5.0),\n",
    "arrowprops=dict(arrowstyle='->'))\n",
    "d2l.plt.xlabel('x')\n",
    "d2l.plt.ylabel('f(x)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saddle points in higher dimensions are even more insidious, as the example below shows. Consider the function $f(x, y) = x^2 - y^2$. It has its saddle point at $(0,0)$. This is a maximum with respect to $y$ and a minimum with respect to $x$. Moreover, it *looks* like a saddle, which is where this mathematical property got its name.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"150.3pt\" version=\"1.1\" viewBox=\"0 0 265.5 150.3\" width=\"265.5pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 150.3 \n",
       "L 265.5 150.3 \n",
       "L 265.5 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"patch_2\">\n",
       "   <path d=\"M 7.2 143.1 \n",
       "L 258.3 143.1 \n",
       "L 258.3 7.2 \n",
       "L 7.2 7.2 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "  </g>\n",
       "  <g id=\"pane3d_1\">\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.487185 113.070889 \n",
       "L 112.353319 79.751289 \n",
       "L 111.156824 16.083 \n",
       "L 35.283682 46.019976 \n",
       "\" style=\"fill:#f2f2f2;opacity:0.5;stroke:#f2f2f2;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"pane3d_2\">\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 112.353319 79.751289 \n",
       "L 228.769699 98.379096 \n",
       "L 233.640421 32.790916 \n",
       "L 111.156824 16.083 \n",
       "\" style=\"fill:#e6e6e6;opacity:0.5;stroke:#e6e6e6;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"pane3d_3\">\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.487185 113.070889 \n",
       "L 162.972947 134.766006 \n",
       "L 228.769699 98.379096 \n",
       "L 112.353319 79.751289 \n",
       "\" style=\"fill:#ececec;opacity:0.5;stroke:#ececec;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axis3d_1\">\n",
       "   <g id=\"line2d_1\">\n",
       "    <path d=\"M 40.487185 113.070889 \n",
       "L 162.972947 134.766006 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_1\">\n",
       "    <!-- x -->\n",
       "    <defs>\n",
       "     <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-120\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(79.250785 151.909667)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-120\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"Line3DCollection_1\">\n",
       "    <path d=\"M 47.954708 114.393564 \n",
       "L 119.473198 80.890542 \n",
       "L 118.632462 17.102748 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "    <path d=\"M 100.33065 123.670578 \n",
       "L 169.328996 88.867978 \n",
       "L 171.035205 24.250975 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "    <path d=\"M 154.838065 133.325126 \n",
       "L 221.062087 97.145799 \n",
       "L 225.514604 31.682478 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"xtick_1\">\n",
       "    <g id=\"line2d_2\">\n",
       "     <path d=\"M 48.570877 114.104918 \n",
       "L 46.720073 114.971932 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_2\">\n",
       "     <!-- −1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-8722\"/>\n",
       "      <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(32.285551 133.414626)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "      <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"xtick_2\">\n",
       "    <g id=\"line2d_3\">\n",
       "     <path d=\"M 100.925952 123.370309 \n",
       "L 99.13778 124.272259 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_3\">\n",
       "     <!-- 0 -->\n",
       "     <defs>\n",
       "      <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(88.843524 142.948134)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"xtick_3\">\n",
       "    <g id=\"line2d_4\">\n",
       "     <path d=\"M 155.410277 133.012517 \n",
       "L 153.69142 133.951558 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- 1 -->\n",
       "     <g transform=\"translate(143.349377 152.87081)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axis3d_2\">\n",
       "   <g id=\"line2d_5\">\n",
       "    <path d=\"M 228.769699 98.379096 \n",
       "L 162.972947 134.766006 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_5\">\n",
       "    <!-- y -->\n",
       "    <defs>\n",
       "     <path d=\"M 32.171875 -5.078125 \n",
       "Q 28.375 -14.84375 24.75 -17.8125 \n",
       "Q 21.140625 -20.796875 15.09375 -20.796875 \n",
       "L 7.90625 -20.796875 \n",
       "L 7.90625 -13.28125 \n",
       "L 13.1875 -13.28125 \n",
       "Q 16.890625 -13.28125 18.9375 -11.515625 \n",
       "Q 21 -9.765625 23.484375 -3.21875 \n",
       "L 25.09375 0.875 \n",
       "L 2.984375 54.6875 \n",
       "L 12.5 54.6875 \n",
       "L 29.59375 11.921875 \n",
       "L 46.6875 54.6875 \n",
       "L 56.203125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-121\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(224.303284 139.624581)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-121\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"Line3DCollection_2\">\n",
       "    <path d=\"M 40.478971 43.970091 \n",
       "L 45.390254 110.797659 \n",
       "L 167.475981 132.275739 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "    <path d=\"M 74.707339 30.464735 \n",
       "L 77.75884 95.790474 \n",
       "L 197.151948 115.86434 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "    <path d=\"M 106.671251 17.852856 \n",
       "L 108.089022 81.728363 \n",
       "L 224.877837 100.531372 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"xtick_4\">\n",
       "    <g id=\"line2d_6\">\n",
       "     <path d=\"M 166.454231 132.095986 \n",
       "L 169.521765 132.635646 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- −1 -->\n",
       "     <g transform=\"translate(174.48021 149.869862)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "      <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"xtick_5\">\n",
       "    <g id=\"line2d_7\">\n",
       "     <path d=\"M 196.154229 115.696591 \n",
       "L 199.149539 116.2002 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_7\">\n",
       "     <!-- 0 -->\n",
       "     <g transform=\"translate(207.834516 133.132167)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"xtick_6\">\n",
       "    <g id=\"line2d_8\">\n",
       "     <path d=\"M 223.90325 100.374464 \n",
       "L 226.829048 100.845517 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_8\">\n",
       "     <!-- 1 -->\n",
       "     <g transform=\"translate(235.083983 117.493501)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axis3d_3\">\n",
       "   <g id=\"line2d_9\">\n",
       "    <path d=\"M 228.769699 98.379096 \n",
       "L 233.640421 32.790916 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"Line3DCollection_3\">\n",
       "    <path d=\"M 228.862339 97.131618 \n",
       "L 112.330513 78.537737 \n",
       "L 40.388382 111.797743 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "    <path d=\"M 231.142669 66.425148 \n",
       "L 111.769747 48.698043 \n",
       "L 37.954325 80.433148 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "    <path d=\"M 233.537992 34.170208 \n",
       "L 111.181932 17.419065 \n",
       "L 35.393294 47.432407 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"xtick_7\">\n",
       "    <g id=\"line2d_10\">\n",
       "     <path d=\"M 227.890048 96.976479 \n",
       "L 230.808945 97.44222 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- −1 -->\n",
       "     <g transform=\"translate(242.240836 103.115424)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "      <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"xtick_8\">\n",
       "    <g id=\"line2d_11\">\n",
       "     <path d=\"M 230.145664 66.277091 \n",
       "L 233.138804 66.721578 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- 0 -->\n",
       "     <g transform=\"translate(249.211018 72.482787)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"xtick_9\">\n",
       "    <g id=\"line2d_12\">\n",
       "     <path d=\"M 232.514986 34.030153 \n",
       "L 235.586238 34.450623 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- 1 -->\n",
       "     <g transform=\"translate(252.131033 40.311251)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"line2d_13\">\n",
       "    <defs>\n",
       "     <path d=\"M -3 3 \n",
       "L 3 -3 \n",
       "M -3 -3 \n",
       "L 3 3 \n",
       "\" id=\"m5b88f9e2f5\" style=\"stroke:#ff0000;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pdebd63276f)\">\n",
       "     <use style=\"fill:#ff0000;stroke:#ff0000;\" x=\"136.143243\" xlink:href=\"#m5b88f9e2f5\" y=\"73.313514\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"Line3DCollection_4\">\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 50.633789 79.513765 \n",
       "L 51.233211 77.943768 \n",
       "L 51.83472 76.39856 \n",
       "L 52.438282 74.878259 \n",
       "L 53.043861 73.382977 \n",
       "L 53.651421 71.912824 \n",
       "L 54.260927 70.467907 \n",
       "L 54.872343 69.048331 \n",
       "L 55.485634 67.654196 \n",
       "L 56.100761 66.285601 \n",
       "L 56.71769 64.942641 \n",
       "L 57.336382 63.625407 \n",
       "L 57.956801 62.333989 \n",
       "L 58.578909 61.068472 \n",
       "L 59.202668 59.82894 \n",
       "L 59.828042 58.61547 \n",
       "L 60.454991 57.428141 \n",
       "L 61.083477 56.267024 \n",
       "L 61.713463 55.13219 \n",
       "L 62.344909 54.023705 \n",
       "L 62.977777 52.941634 \n",
       "L 63.612028 51.886035 \n",
       "L 64.247622 50.856967 \n",
       "L 64.884521 49.854483 \n",
       "L 65.522685 48.878633 \n",
       "L 66.162075 47.929464 \n",
       "L 66.802651 47.007021 \n",
       "L 67.444373 46.111345 \n",
       "L 68.087201 45.242471 \n",
       "L 68.731096 44.400435 \n",
       "L 69.376017 43.585267 \n",
       "L 70.021925 42.796994 \n",
       "L 70.668779 42.035641 \n",
       "L 71.316538 41.301228 \n",
       "L 71.965163 40.593773 \n",
       "L 72.614614 39.91329 \n",
       "L 73.264849 39.25979 \n",
       "L 73.915828 38.633281 \n",
       "L 74.567511 38.033766 \n",
       "L 75.219858 37.461247 \n",
       "L 75.872827 36.915722 \n",
       "L 76.526379 36.397185 \n",
       "L 77.180473 35.905627 \n",
       "L 77.835068 35.441036 \n",
       "L 78.490124 35.003397 \n",
       "L 79.1456 34.592691 \n",
       "L 79.801457 34.208897 \n",
       "L 80.457654 33.85199 \n",
       "L 81.114151 33.521941 \n",
       "L 81.770906 33.218719 \n",
       "L 82.427882 32.94229 \n",
       "L 83.085036 32.692617 \n",
       "L 83.74233 32.469658 \n",
       "L 84.399723 32.27337 \n",
       "L 85.057176 32.103705 \n",
       "L 85.714649 31.960615 \n",
       "L 86.372102 31.844045 \n",
       "L 87.029497 31.753941 \n",
       "L 87.686793 31.690242 \n",
       "L 88.343952 31.652887 \n",
       "L 89.000935 31.641811 \n",
       "L 89.657703 31.656946 \n",
       "L 90.314216 31.698222 \n",
       "L 90.970438 31.765564 \n",
       "L 91.626329 31.858897 \n",
       "L 92.281851 31.978141 \n",
       "L 92.936967 32.123214 \n",
       "L 93.591638 32.294031 \n",
       "L 94.245828 32.490506 \n",
       "L 94.899498 32.712547 \n",
       "L 95.552611 32.960061 \n",
       "L 96.205131 33.232954 \n",
       "L 96.857022 33.531127 \n",
       "L 97.508246 33.85448 \n",
       "L 98.158767 34.202909 \n",
       "L 98.808549 34.576309 \n",
       "L 99.457557 34.974572 \n",
       "L 100.105756 35.397587 \n",
       "L 100.753109 35.845241 \n",
       "L 101.399583 36.317419 \n",
       "L 102.045142 36.814003 \n",
       "L 102.689752 37.334873 \n",
       "L 103.33338 37.879908 \n",
       "L 103.97599 38.448984 \n",
       "L 104.61755 39.041972 \n",
       "L 105.258027 39.658746 \n",
       "L 105.897387 40.299174 \n",
       "L 106.535598 40.963123 \n",
       "L 107.172628 41.650459 \n",
       "L 107.808445 42.361045 \n",
       "L 108.443017 43.094743 \n",
       "L 109.076312 43.851412 \n",
       "L 109.7083 44.630909 \n",
       "L 110.338951 45.433091 \n",
       "L 110.968232 46.257811 \n",
       "L 111.596116 47.104922 \n",
       "L 112.222571 47.974274 \n",
       "L 112.847569 48.865717 \n",
       "L 113.471081 49.779097 \n",
       "L 114.093078 50.714261 \n",
       "L 114.713531 51.671053 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 61.857599 92.726574 \n",
       "L 62.457449 91.168715 \n",
       "L 63.059149 89.63526 \n",
       "L 63.662664 88.126322 \n",
       "L 64.267959 86.642014 \n",
       "L 64.874999 85.182444 \n",
       "L 65.483747 83.747717 \n",
       "L 66.094169 82.337936 \n",
       "L 66.706227 80.953202 \n",
       "L 67.319886 79.59361 \n",
       "L 67.935109 78.259255 \n",
       "L 68.551859 76.950227 \n",
       "L 69.1701 75.666614 \n",
       "L 69.789794 74.408502 \n",
       "L 70.410904 73.175971 \n",
       "L 71.033392 71.969101 \n",
       "L 71.657221 70.787966 \n",
       "L 72.282353 69.632639 \n",
       "L 72.90875 68.50319 \n",
       "L 73.536374 67.399684 \n",
       "L 74.165186 66.322185 \n",
       "L 74.795148 65.270751 \n",
       "L 75.426222 64.24544 \n",
       "L 76.058368 63.246305 \n",
       "L 76.691549 62.273396 \n",
       "L 77.325724 61.326761 \n",
       "L 77.960856 60.406442 \n",
       "L 78.596905 59.51248 \n",
       "L 79.233831 58.644913 \n",
       "L 79.871597 57.803775 \n",
       "L 80.510161 56.989096 \n",
       "L 81.149486 56.200904 \n",
       "L 81.789531 55.439223 \n",
       "L 82.430258 54.704075 \n",
       "L 83.071626 53.995477 \n",
       "L 83.713597 53.313444 \n",
       "L 84.35613 52.657987 \n",
       "L 84.999187 52.029115 \n",
       "L 85.642727 51.426832 \n",
       "L 86.286712 50.85114 \n",
       "L 86.931101 50.302038 \n",
       "L 87.575856 49.77952 \n",
       "L 88.220937 49.28358 \n",
       "L 88.866304 48.814205 \n",
       "L 89.511918 48.371382 \n",
       "L 90.15774 47.955092 \n",
       "L 90.803731 47.565317 \n",
       "L 91.449852 47.202031 \n",
       "L 92.096063 46.865208 \n",
       "L 92.742325 46.554818 \n",
       "L 93.3886 46.270827 \n",
       "L 94.034849 46.0132 \n",
       "L 94.681033 45.781898 \n",
       "L 95.327114 45.576878 \n",
       "L 95.973053 45.398095 \n",
       "L 96.618811 45.245501 \n",
       "L 97.264351 45.119044 \n",
       "L 97.909635 45.01867 \n",
       "L 98.554625 44.944322 \n",
       "L 99.199283 44.89594 \n",
       "L 99.843571 44.873461 \n",
       "L 100.487452 44.876818 \n",
       "L 101.13089 44.905945 \n",
       "L 101.773846 44.960768 \n",
       "L 102.416285 45.041214 \n",
       "L 103.058168 45.147205 \n",
       "L 103.699461 45.278662 \n",
       "L 104.340127 45.435503 \n",
       "L 104.98013 45.617641 \n",
       "L 105.619434 45.82499 \n",
       "L 106.258004 46.057459 \n",
       "L 106.895803 46.314954 \n",
       "L 107.532798 46.597381 \n",
       "L 108.168954 46.904642 \n",
       "L 108.804235 47.236635 \n",
       "L 109.438608 47.593257 \n",
       "L 110.072038 47.974404 \n",
       "L 110.704492 48.379968 \n",
       "L 111.335937 48.809837 \n",
       "L 111.966338 49.263901 \n",
       "L 112.595663 49.742044 \n",
       "L 113.22388 50.244148 \n",
       "L 113.850956 50.770096 \n",
       "L 114.476859 51.319765 \n",
       "L 115.101556 51.893032 \n",
       "L 115.725018 52.489772 \n",
       "L 116.347212 53.109856 \n",
       "L 116.968108 53.753156 \n",
       "L 117.587675 54.419539 \n",
       "L 118.205883 55.108872 \n",
       "L 118.822702 55.821019 \n",
       "L 119.438103 56.555843 \n",
       "L 120.052056 57.313206 \n",
       "L 120.664532 58.092965 \n",
       "L 121.275504 58.894978 \n",
       "L 121.884941 59.719101 \n",
       "L 122.492818 60.565188 \n",
       "L 123.099106 61.433091 \n",
       "L 123.703777 62.32266 \n",
       "L 124.306806 63.233746 \n",
       "L 124.908166 64.166195 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 72.840106 103.309598 \n",
       "L 73.441513 101.759826 \n",
       "L 74.044545 100.234174 \n",
       "L 74.649165 98.732756 \n",
       "L 75.255339 97.255681 \n",
       "L 75.863031 95.803057 \n",
       "L 76.472205 94.374988 \n",
       "L 77.082825 92.971577 \n",
       "L 77.694857 91.592923 \n",
       "L 78.308262 90.239121 \n",
       "L 78.923004 88.910265 \n",
       "L 79.539048 87.606444 \n",
       "L 80.156355 86.327747 \n",
       "L 80.77489 85.074256 \n",
       "L 81.394614 83.846053 \n",
       "L 82.015491 82.643216 \n",
       "L 82.637483 81.46582 \n",
       "L 83.260552 80.313937 \n",
       "L 83.88466 79.187635 \n",
       "L 84.509771 78.086981 \n",
       "L 85.135845 77.012037 \n",
       "L 85.762844 75.962862 \n",
       "L 86.390731 74.939513 \n",
       "L 87.019467 73.942043 \n",
       "L 87.649014 72.970503 \n",
       "L 88.279334 72.024937 \n",
       "L 88.910387 71.105392 \n",
       "L 89.542135 70.211906 \n",
       "L 90.17454 69.344518 \n",
       "L 90.807562 68.503261 \n",
       "L 91.441164 67.688167 \n",
       "L 92.075307 66.899263 \n",
       "L 92.709951 66.136573 \n",
       "L 93.345058 65.40012 \n",
       "L 93.980589 64.689921 \n",
       "L 94.616505 64.005991 \n",
       "L 95.252768 63.348343 \n",
       "L 95.889339 62.716984 \n",
       "L 96.526179 62.11192 \n",
       "L 97.16325 61.533153 \n",
       "L 97.800512 60.980683 \n",
       "L 98.437927 60.454505 \n",
       "L 99.075457 59.954612 \n",
       "L 99.713063 59.480994 \n",
       "L 100.350707 59.033637 \n",
       "L 100.98835 58.612524 \n",
       "L 101.625954 58.217637 \n",
       "L 102.263481 57.848951 \n",
       "L 102.900893 57.506441 \n",
       "L 103.538152 57.190079 \n",
       "L 104.175221 56.899831 \n",
       "L 104.81206 56.635663 \n",
       "L 105.448634 56.397537 \n",
       "L 106.084904 56.185411 \n",
       "L 106.720833 55.999242 \n",
       "L 107.356384 55.838983 \n",
       "L 107.99152 55.704583 \n",
       "L 108.626204 55.595989 \n",
       "L 109.260399 55.513146 \n",
       "L 109.89407 55.455995 \n",
       "L 110.527179 55.424474 \n",
       "L 111.159691 55.41852 \n",
       "L 111.79157 55.438064 \n",
       "L 112.42278 55.483037 \n",
       "L 113.053285 55.553365 \n",
       "L 113.683051 55.648975 \n",
       "L 114.312041 55.769787 \n",
       "L 114.940222 55.91572 \n",
       "L 115.567559 56.086692 \n",
       "L 116.194017 56.282615 \n",
       "L 116.819563 56.503402 \n",
       "L 117.444162 56.748961 \n",
       "L 118.067781 57.019198 \n",
       "L 118.690386 57.314017 \n",
       "L 119.311944 57.63332 \n",
       "L 119.932423 57.977005 \n",
       "L 120.55179 58.344968 \n",
       "L 121.170013 58.737105 \n",
       "L 121.787059 59.153306 \n",
       "L 122.402898 59.593461 \n",
       "L 123.017497 60.057458 \n",
       "L 123.630826 60.545182 \n",
       "L 124.242853 61.056515 \n",
       "L 124.853549 61.591338 \n",
       "L 125.462882 62.149529 \n",
       "L 126.070824 62.730967 \n",
       "L 126.677345 63.335523 \n",
       "L 127.282414 63.963073 \n",
       "L 127.886005 64.613485 \n",
       "L 128.488087 65.286628 \n",
       "L 129.088633 65.98237 \n",
       "L 129.687614 66.700575 \n",
       "L 130.285004 67.441106 \n",
       "L 130.880775 68.203825 \n",
       "L 131.474899 68.988591 \n",
       "L 132.067351 69.795262 \n",
       "L 132.658105 70.623694 \n",
       "L 133.247133 71.473742 \n",
       "L 133.834411 72.345259 \n",
       "L 134.419915 73.238097 \n",
       "L 135.003618 74.152106 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 83.649779 111.350964 \n",
       "L 84.253893 109.805342 \n",
       "L 84.859414 108.283655 \n",
       "L 85.466305 106.786015 \n",
       "L 86.074532 105.312532 \n",
       "L 86.684058 103.863314 \n",
       "L 87.294848 102.438464 \n",
       "L 87.906866 101.038084 \n",
       "L 88.520075 99.662273 \n",
       "L 89.134439 98.311126 \n",
       "L 89.749921 96.984736 \n",
       "L 90.366484 95.683192 \n",
       "L 90.984092 94.406583 \n",
       "L 91.602708 93.15499 \n",
       "L 92.222294 91.928496 \n",
       "L 92.842813 90.727178 \n",
       "L 93.464227 89.55111 \n",
       "L 94.086499 88.400366 \n",
       "L 94.709592 87.275012 \n",
       "L 95.333467 86.175115 \n",
       "L 95.958087 85.100738 \n",
       "L 96.583413 84.051939 \n",
       "L 97.209409 83.028775 \n",
       "L 97.836035 82.0313 \n",
       "L 98.463253 81.059563 \n",
       "L 99.091027 80.113611 \n",
       "L 99.719316 79.193488 \n",
       "L 100.348084 78.299234 \n",
       "L 100.977291 77.430887 \n",
       "L 101.606899 76.588482 \n",
       "L 102.236871 75.772049 \n",
       "L 102.867167 74.981617 \n",
       "L 103.497751 74.217209 \n",
       "L 104.128582 73.478849 \n",
       "L 104.759623 72.766554 \n",
       "L 105.390836 72.080339 \n",
       "L 106.022183 71.420217 \n",
       "L 106.653625 70.786196 \n",
       "L 107.285124 70.178283 \n",
       "L 107.916642 69.59648 \n",
       "L 108.548142 69.040787 \n",
       "L 109.179584 68.511199 \n",
       "L 109.810933 68.007711 \n",
       "L 110.442149 67.530312 \n",
       "L 111.073195 67.07899 \n",
       "L 111.704033 66.653728 \n",
       "L 112.334627 66.254507 \n",
       "L 112.964938 65.881305 \n",
       "L 113.59493 65.534096 \n",
       "L 114.224565 65.212853 \n",
       "L 114.853807 64.917544 \n",
       "L 115.482618 64.648134 \n",
       "L 116.110962 64.404586 \n",
       "L 116.738803 64.18686 \n",
       "L 117.366104 63.994913 \n",
       "L 117.99283 63.828697 \n",
       "L 118.618943 63.688165 \n",
       "L 119.244409 63.573263 \n",
       "L 119.869192 63.483937 \n",
       "L 120.493256 63.420129 \n",
       "L 121.116566 63.381779 \n",
       "L 121.739087 63.368822 \n",
       "L 122.360784 63.381193 \n",
       "L 122.981624 63.418822 \n",
       "L 123.601571 63.481638 \n",
       "L 124.220591 63.569567 \n",
       "L 124.838651 63.682531 \n",
       "L 125.455717 63.82045 \n",
       "L 126.071756 63.983242 \n",
       "L 126.686734 64.170822 \n",
       "L 127.300619 64.383102 \n",
       "L 127.913377 64.619992 \n",
       "L 128.524978 64.8814 \n",
       "L 129.135388 65.16723 \n",
       "L 129.744577 65.477386 \n",
       "L 130.352511 65.811766 \n",
       "L 130.959161 66.170269 \n",
       "L 131.564496 66.55279 \n",
       "L 132.168484 66.959222 \n",
       "L 132.771096 67.389456 \n",
       "L 133.372302 67.84338 \n",
       "L 133.972071 68.320881 \n",
       "L 134.570375 68.821842 \n",
       "L 135.167184 69.346146 \n",
       "L 135.76247 69.893673 \n",
       "L 136.356204 70.464301 \n",
       "L 136.948358 71.057904 \n",
       "L 137.538905 71.674358 \n",
       "L 138.127818 72.313534 \n",
       "L 138.715068 72.975302 \n",
       "L 139.300629 73.659531 \n",
       "L 139.884475 74.366086 \n",
       "L 140.46658 75.094831 \n",
       "L 141.046919 75.845631 \n",
       "L 141.625465 76.618345 \n",
       "L 142.202193 77.412833 \n",
       "L 142.77708 78.228953 \n",
       "L 143.350101 79.06656 \n",
       "L 143.921232 79.92551 \n",
       "L 144.490449 80.805655 \n",
       "L 145.057729 81.706847 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 94.352434 116.907381 \n",
       "L 94.960437 115.362026 \n",
       "L 95.569635 113.840516 \n",
       "L 96.179991 112.342964 \n",
       "L 96.79147 110.86948 \n",
       "L 97.404036 109.420172 \n",
       "L 98.017651 107.995143 \n",
       "L 98.63228 106.594496 \n",
       "L 99.247887 105.218328 \n",
       "L 99.864433 103.866735 \n",
       "L 100.481884 102.539811 \n",
       "L 101.1002 101.237644 \n",
       "L 101.719347 99.960322 \n",
       "L 102.339285 98.70793 \n",
       "L 102.959978 97.480547 \n",
       "L 103.581388 96.278252 \n",
       "L 104.203478 95.101119 \n",
       "L 104.82621 93.949221 \n",
       "L 105.449546 92.822626 \n",
       "L 106.073449 91.7214 \n",
       "L 106.697881 90.645606 \n",
       "L 107.322804 89.595304 \n",
       "L 107.94818 88.570549 \n",
       "L 108.57397 87.571395 \n",
       "L 109.200138 86.597892 \n",
       "L 109.826644 85.650088 \n",
       "L 110.453451 84.728027 \n",
       "L 111.080521 83.831748 \n",
       "L 111.707816 82.961291 \n",
       "L 112.335297 82.116689 \n",
       "L 112.962927 81.297973 \n",
       "L 113.590667 80.505173 \n",
       "L 114.21848 79.738312 \n",
       "L 114.846327 78.997414 \n",
       "L 115.474172 78.282496 \n",
       "L 116.101975 77.593574 \n",
       "L 116.729699 76.93066 \n",
       "L 117.357306 76.293764 \n",
       "L 117.984759 75.682892 \n",
       "L 118.612021 75.098047 \n",
       "L 119.239052 74.539229 \n",
       "L 119.865817 74.006434 \n",
       "L 120.492278 73.499655 \n",
       "L 121.118398 73.018884 \n",
       "L 121.744139 72.564107 \n",
       "L 122.369466 72.13531 \n",
       "L 122.99434 71.732472 \n",
       "L 123.618725 71.355573 \n",
       "L 124.242586 71.004587 \n",
       "L 124.865884 70.679486 \n",
       "L 125.488586 70.380239 \n",
       "L 126.110653 70.106812 \n",
       "L 126.732051 69.859168 \n",
       "L 127.352743 69.637268 \n",
       "L 127.972695 69.441068 \n",
       "L 128.591871 69.270522 \n",
       "L 129.210235 69.125581 \n",
       "L 129.827754 69.006194 \n",
       "L 130.444391 68.912307 \n",
       "L 131.060114 68.843861 \n",
       "L 131.674887 68.800797 \n",
       "L 132.288677 68.783051 \n",
       "L 132.90145 68.790558 \n",
       "L 133.513171 68.823248 \n",
       "L 134.123809 68.881052 \n",
       "L 134.73333 68.963894 \n",
       "L 135.341701 69.071698 \n",
       "L 135.94889 69.204384 \n",
       "L 136.554864 69.361871 \n",
       "L 137.159592 69.544074 \n",
       "L 137.763041 69.750906 \n",
       "L 138.36518 69.982277 \n",
       "L 138.965979 70.238096 \n",
       "L 139.565407 70.518266 \n",
       "L 140.163432 70.822692 \n",
       "L 140.760024 71.151274 \n",
       "L 141.355154 71.50391 \n",
       "L 141.948793 71.880496 \n",
       "L 142.54091 72.280925 \n",
       "L 143.131476 72.705089 \n",
       "L 143.720464 73.152876 \n",
       "L 144.307844 73.624174 \n",
       "L 144.893589 74.118867 \n",
       "L 145.477671 74.636837 \n",
       "L 146.060062 75.177966 \n",
       "L 146.640736 75.742131 \n",
       "L 147.219666 76.329209 \n",
       "L 147.796825 76.939074 \n",
       "L 148.372188 77.571598 \n",
       "L 148.945728 78.226653 \n",
       "L 149.517421 78.904106 \n",
       "L 150.087241 79.603824 \n",
       "L 150.655165 80.325674 \n",
       "L 151.221167 81.069516 \n",
       "L 151.785224 81.835215 \n",
       "L 152.347312 82.622628 \n",
       "L 152.907408 83.431614 \n",
       "L 153.46549 84.262031 \n",
       "L 154.021535 85.113733 \n",
       "L 154.57552 85.986573 \n",
       "L 155.127425 86.880404 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 105.012571 120.005446 \n",
       "L 105.625693 118.456473 \n",
       "L 106.239801 116.931352 \n",
       "L 106.854858 115.430196 \n",
       "L 107.470828 113.953115 \n",
       "L 108.087673 112.500217 \n",
       "L 108.705358 111.071607 \n",
       "L 109.323845 109.667387 \n",
       "L 109.943098 108.287655 \n",
       "L 110.563078 106.932509 \n",
       "L 111.183749 105.602041 \n",
       "L 111.805073 104.296342 \n",
       "L 112.427014 103.015499 \n",
       "L 113.049532 101.759596 \n",
       "L 113.672591 100.528715 \n",
       "L 114.296153 99.322935 \n",
       "L 114.92018 98.142331 \n",
       "L 115.544634 96.986974 \n",
       "L 116.169477 95.856934 \n",
       "L 116.794672 94.752278 \n",
       "L 117.42018 93.673068 \n",
       "L 118.045963 92.619365 \n",
       "L 118.671983 91.591224 \n",
       "L 119.298202 90.588701 \n",
       "L 119.924583 89.611845 \n",
       "L 120.551086 88.660704 \n",
       "L 121.177674 87.735322 \n",
       "L 121.804309 86.83574 \n",
       "L 122.430953 85.961997 \n",
       "L 123.057568 85.114127 \n",
       "L 123.684116 84.292161 \n",
       "L 124.310559 83.496128 \n",
       "L 124.936859 82.726054 \n",
       "L 125.562979 81.981961 \n",
       "L 126.18888 81.263867 \n",
       "L 126.814526 80.571788 \n",
       "L 127.439878 79.905737 \n",
       "L 128.064899 79.265724 \n",
       "L 128.689552 78.651754 \n",
       "L 129.3138 78.063831 \n",
       "L 129.937605 77.501954 \n",
       "L 130.56093 76.966121 \n",
       "L 131.183739 76.456325 \n",
       "L 131.805995 75.972557 \n",
       "L 132.427661 75.514804 \n",
       "L 133.048701 75.083051 \n",
       "L 133.669078 74.677278 \n",
       "L 134.288757 74.297464 \n",
       "L 134.907701 73.943585 \n",
       "L 135.525875 73.615612 \n",
       "L 136.143243 73.313514 \n",
       "L 136.75977 73.037257 \n",
       "L 137.37542 72.786804 \n",
       "L 137.990159 72.562115 \n",
       "L 138.603952 72.363148 \n",
       "L 139.216764 72.189856 \n",
       "L 139.82856 72.04219 \n",
       "L 140.439307 71.920099 \n",
       "L 141.048972 71.823529 \n",
       "L 141.657519 71.75242 \n",
       "L 142.264916 71.706715 \n",
       "L 142.87113 71.686348 \n",
       "L 143.476127 71.691254 \n",
       "L 144.079876 71.721366 \n",
       "L 144.682343 71.77661 \n",
       "L 145.283498 71.856913 \n",
       "L 145.883307 71.962198 \n",
       "L 146.481739 72.092385 \n",
       "L 147.078763 72.247393 \n",
       "L 147.674349 72.427137 \n",
       "L 148.268465 72.631529 \n",
       "L 148.861081 72.86048 \n",
       "L 149.452167 73.113897 \n",
       "L 150.041694 73.391685 \n",
       "L 150.629631 73.693748 \n",
       "L 151.215951 74.019985 \n",
       "L 151.800623 74.370294 \n",
       "L 152.38362 74.744572 \n",
       "L 152.964913 75.14271 \n",
       "L 153.544475 75.564601 \n",
       "L 154.122278 76.010133 \n",
       "L 154.698296 76.479193 \n",
       "L 155.2725 76.971665 \n",
       "L 155.844866 77.48743 \n",
       "L 156.415366 78.026371 \n",
       "L 156.983975 78.588363 \n",
       "L 157.550667 79.173284 \n",
       "L 158.115419 79.781008 \n",
       "L 158.678204 80.411407 \n",
       "L 159.238999 81.06435 \n",
       "L 159.79778 81.739707 \n",
       "L 160.354522 82.437343 \n",
       "L 160.909204 83.157124 \n",
       "L 161.461802 83.898912 \n",
       "L 162.012293 84.662568 \n",
       "L 162.560656 85.447953 \n",
       "L 163.106868 86.254924 \n",
       "L 163.650909 87.083337 \n",
       "L 164.192758 87.933047 \n",
       "L 164.732393 88.803907 \n",
       "L 165.269795 89.695769 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 115.694682 120.642048 \n",
       "L 116.314215 119.085516 \n",
       "L 116.934525 117.552939 \n",
       "L 117.555576 116.044429 \n",
       "L 118.17733 114.5601 \n",
       "L 118.799749 113.100058 \n",
       "L 119.422797 111.664411 \n",
       "L 120.046435 110.253261 \n",
       "L 120.670626 108.866708 \n",
       "L 121.295333 107.50485 \n",
       "L 121.920516 106.167781 \n",
       "L 122.54614 104.855591 \n",
       "L 123.172164 103.56837 \n",
       "L 123.798552 102.306202 \n",
       "L 124.425265 101.06917 \n",
       "L 125.052265 99.857353 \n",
       "L 125.679514 98.670826 \n",
       "L 126.306973 97.509664 \n",
       "L 126.934604 96.373935 \n",
       "L 127.56237 95.263708 \n",
       "L 128.190231 94.179044 \n",
       "L 128.818149 93.120006 \n",
       "L 129.446086 92.08665 \n",
       "L 130.074003 91.07903 \n",
       "L 130.701864 90.097198 \n",
       "L 131.329628 89.141202 \n",
       "L 131.957258 88.211086 \n",
       "L 132.584716 87.306892 \n",
       "L 133.211963 86.428659 \n",
       "L 133.838962 85.576421 \n",
       "L 134.465675 84.75021 \n",
       "L 135.092063 83.950055 \n",
       "L 135.718089 83.175982 \n",
       "L 136.343716 82.428014 \n",
       "L 136.968904 81.706168 \n",
       "L 137.593618 81.010462 \n",
       "L 138.217819 80.340907 \n",
       "L 138.841471 79.697514 \n",
       "L 139.464535 79.080289 \n",
       "L 140.086976 78.489234 \n",
       "L 140.708756 77.924351 \n",
       "L 141.329838 77.385635 \n",
       "L 141.950185 76.873081 \n",
       "L 142.569762 76.386679 \n",
       "L 143.188533 75.926416 \n",
       "L 143.80646 75.492277 \n",
       "L 144.423508 75.084242 \n",
       "L 145.039642 74.70229 \n",
       "L 145.654825 74.346395 \n",
       "L 146.269023 74.016529 \n",
       "L 146.882201 73.712662 \n",
       "L 147.494323 73.434758 \n",
       "L 148.105355 73.182781 \n",
       "L 148.715263 72.956689 \n",
       "L 149.324012 72.75644 \n",
       "L 149.931568 72.581988 \n",
       "L 150.537898 72.433282 \n",
       "L 151.142968 72.310271 \n",
       "L 151.746745 72.2129 \n",
       "L 152.349196 72.141111 \n",
       "L 152.950288 72.094842 \n",
       "L 153.54999 72.074031 \n",
       "L 154.148268 72.078609 \n",
       "L 154.745092 72.108509 \n",
       "L 155.340429 72.163658 \n",
       "L 155.934248 72.243981 \n",
       "L 156.526519 72.349401 \n",
       "L 157.11721 72.479837 \n",
       "L 157.706292 72.635207 \n",
       "L 158.293734 72.815425 \n",
       "L 158.879507 73.020402 \n",
       "L 159.463581 73.250049 \n",
       "L 160.045927 73.504272 \n",
       "L 160.626517 73.782975 \n",
       "L 161.205322 74.08606 \n",
       "L 161.782314 74.413427 \n",
       "L 162.357466 74.764973 \n",
       "L 162.930749 75.140592 \n",
       "L 163.502138 75.540176 \n",
       "L 164.071605 75.963616 \n",
       "L 164.639124 76.4108 \n",
       "L 165.204669 76.881612 \n",
       "L 165.768215 77.375936 \n",
       "L 166.329736 77.893654 \n",
       "L 166.889207 78.434645 \n",
       "L 167.446604 78.998785 \n",
       "L 168.001904 79.585949 \n",
       "L 168.555081 80.196011 \n",
       "L 169.106113 80.828841 \n",
       "L 169.654977 81.484308 \n",
       "L 170.201649 82.16228 \n",
       "L 170.746109 82.862623 \n",
       "L 171.288333 83.585198 \n",
       "L 171.828302 84.329869 \n",
       "L 172.365992 85.096494 \n",
       "L 172.901384 85.884934 \n",
       "L 173.434458 86.695043 \n",
       "L 173.965193 87.526677 \n",
       "L 174.49357 88.379691 \n",
       "L 175.01957 89.253934 \n",
       "L 175.543173 90.149259 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 126.464565 118.783854 \n",
       "L 127.091879 117.215706 \n",
       "L 127.719762 115.671713 \n",
       "L 128.348173 114.15199 \n",
       "L 128.977076 112.656652 \n",
       "L 129.606432 111.185808 \n",
       "L 130.236202 109.739565 \n",
       "L 130.866349 108.318029 \n",
       "L 131.496834 106.9213 \n",
       "L 132.127617 105.549478 \n",
       "L 132.758662 104.202657 \n",
       "L 133.389928 102.880931 \n",
       "L 134.021378 101.58439 \n",
       "L 134.652973 100.313119 \n",
       "L 135.284673 99.067201 \n",
       "L 135.91644 97.846718 \n",
       "L 136.548235 96.651747 \n",
       "L 137.18002 95.482361 \n",
       "L 137.811755 94.338631 \n",
       "L 138.443402 93.220626 \n",
       "L 139.074922 92.128409 \n",
       "L 139.706276 91.062043 \n",
       "L 140.337426 90.021584 \n",
       "L 140.968332 89.00709 \n",
       "L 141.598957 88.01861 \n",
       "L 142.22926 87.056194 \n",
       "L 142.859205 86.119888 \n",
       "L 143.488753 85.209732 \n",
       "L 144.117865 84.325767 \n",
       "L 144.746502 83.468027 \n",
       "L 145.374628 82.636544 \n",
       "L 146.002203 81.831349 \n",
       "L 146.629189 81.052467 \n",
       "L 147.25555 80.299921 \n",
       "L 147.881246 79.573729 \n",
       "L 148.506242 78.873908 \n",
       "L 149.130498 78.200471 \n",
       "L 149.753978 77.553426 \n",
       "L 150.376645 76.932782 \n",
       "L 150.998461 76.33854 \n",
       "L 151.619391 75.770701 \n",
       "L 152.239396 75.229261 \n",
       "L 152.858441 74.714214 \n",
       "L 153.47649 74.225549 \n",
       "L 154.093507 73.763255 \n",
       "L 154.709455 73.327314 \n",
       "L 155.324299 72.917707 \n",
       "L 155.938003 72.534412 \n",
       "L 156.550533 72.177404 \n",
       "L 157.161854 71.846653 \n",
       "L 157.77193 71.542128 \n",
       "L 158.380727 71.263793 \n",
       "L 158.988211 71.011611 \n",
       "L 159.594348 70.78554 \n",
       "L 160.199105 70.585537 \n",
       "L 160.802447 70.411554 \n",
       "L 161.404342 70.26354 \n",
       "L 162.004757 70.141444 \n",
       "L 162.603659 70.045208 \n",
       "L 163.201016 69.974774 \n",
       "L 163.796795 69.930079 \n",
       "L 164.390966 69.911059 \n",
       "L 164.983496 69.917647 \n",
       "L 165.574355 69.94977 \n",
       "L 166.163512 70.007358 \n",
       "L 166.750936 70.090332 \n",
       "L 167.336597 70.198614 \n",
       "L 167.920466 70.332123 \n",
       "L 168.502512 70.490775 \n",
       "L 169.082708 70.674482 \n",
       "L 169.661023 70.883154 \n",
       "L 170.237429 71.116701 \n",
       "L 170.811899 71.375026 \n",
       "L 171.384405 71.658034 \n",
       "L 171.954919 71.965623 \n",
       "L 172.523414 72.297692 \n",
       "L 173.089863 72.654135 \n",
       "L 173.654241 73.034847 \n",
       "L 174.21652 73.439718 \n",
       "L 174.776677 73.868635 \n",
       "L 175.334685 74.321485 \n",
       "L 175.890519 74.798152 \n",
       "L 176.444156 75.298516 \n",
       "L 176.995571 75.822458 \n",
       "L 177.544741 76.369855 \n",
       "L 178.091642 76.94058 \n",
       "L 178.636251 77.534509 \n",
       "L 179.178546 78.151511 \n",
       "L 179.718504 78.791455 \n",
       "L 180.256105 79.454209 \n",
       "L 180.791326 80.139638 \n",
       "L 181.324147 80.847605 \n",
       "L 181.854548 81.57797 \n",
       "L 182.382507 82.330595 \n",
       "L 182.908006 83.105337 \n",
       "L 183.431025 83.902052 \n",
       "L 183.951546 84.720594 \n",
       "L 184.469549 85.560817 \n",
       "L 184.985016 86.422571 \n",
       "L 185.49793 87.305707 \n",
       "L 186.008274 88.210072 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 137.390712 114.365856 \n",
       "L 138.027279 112.781854 \n",
       "L 138.6642 111.222311 \n",
       "L 139.301433 109.687346 \n",
       "L 139.938942 108.177073 \n",
       "L 140.576685 106.691605 \n",
       "L 141.214624 105.231053 \n",
       "L 141.85272 103.795522 \n",
       "L 142.490932 102.385116 \n",
       "L 143.129222 100.999935 \n",
       "L 143.76755 99.640077 \n",
       "L 144.405876 98.305637 \n",
       "L 145.04416 96.996706 \n",
       "L 145.682364 95.713371 \n",
       "L 146.320448 94.455717 \n",
       "L 146.958372 93.223828 \n",
       "L 147.596096 92.01778 \n",
       "L 148.23358 90.837651 \n",
       "L 148.870787 89.683511 \n",
       "L 149.507675 88.55543 \n",
       "L 150.144205 87.453473 \n",
       "L 150.780339 86.377704 \n",
       "L 151.416036 85.328181 \n",
       "L 152.051258 84.30496 \n",
       "L 152.685965 83.308095 \n",
       "L 153.320118 82.337633 \n",
       "L 153.953679 81.393622 \n",
       "L 154.586608 80.476105 \n",
       "L 155.218866 79.585119 \n",
       "L 155.850416 78.720703 \n",
       "L 156.481218 77.882888 \n",
       "L 157.111233 77.071704 \n",
       "L 157.740425 76.287177 \n",
       "L 158.368754 75.52933 \n",
       "L 158.996183 74.798183 \n",
       "L 159.622674 74.093751 \n",
       "L 160.248189 73.416047 \n",
       "L 160.872691 72.765081 \n",
       "L 161.496142 72.140859 \n",
       "L 162.118507 71.543385 \n",
       "L 162.739746 70.972657 \n",
       "L 163.359825 70.428671 \n",
       "L 163.978707 69.911422 \n",
       "L 164.596355 69.420898 \n",
       "L 165.212733 68.957086 \n",
       "L 165.827807 68.519969 \n",
       "L 166.441539 68.109528 \n",
       "L 167.053896 67.725738 \n",
       "L 167.664841 67.368574 \n",
       "L 168.274341 67.038006 \n",
       "L 168.882361 66.734001 \n",
       "L 169.488866 66.456522 \n",
       "L 170.093823 66.205531 \n",
       "L 170.697198 65.980986 \n",
       "L 171.298958 65.782841 \n",
       "L 171.899069 65.611047 \n",
       "L 172.4975 65.465554 \n",
       "L 173.094217 65.346305 \n",
       "L 173.689189 65.253245 \n",
       "L 174.282384 65.186312 \n",
       "L 174.873769 65.145442 \n",
       "L 175.463315 65.130569 \n",
       "L 176.05099 65.141624 \n",
       "L 176.636765 65.178535 \n",
       "L 177.220607 65.241225 \n",
       "L 177.802489 65.329618 \n",
       "L 178.382381 65.443632 \n",
       "L 178.960253 65.583184 \n",
       "L 179.536077 65.748187 \n",
       "L 180.109824 65.938552 \n",
       "L 180.681467 66.154188 \n",
       "L 181.250978 66.394999 \n",
       "L 181.81833 66.66089 \n",
       "L 182.383496 66.951761 \n",
       "L 182.946449 67.267509 \n",
       "L 183.507163 67.60803 \n",
       "L 184.065614 67.973216 \n",
       "L 184.621774 68.362958 \n",
       "L 185.175621 68.777145 \n",
       "L 185.727128 69.215661 \n",
       "L 186.276272 69.67839 \n",
       "L 186.823029 70.165214 \n",
       "L 187.367377 70.67601 \n",
       "L 187.909291 71.210656 \n",
       "L 188.44875 71.769025 \n",
       "L 188.985732 72.350991 \n",
       "L 189.520215 72.956423 \n",
       "L 190.052177 73.585189 \n",
       "L 190.581597 74.237156 \n",
       "L 191.108457 74.912187 \n",
       "L 191.632734 75.610144 \n",
       "L 192.15441 76.330889 \n",
       "L 192.673466 77.074278 \n",
       "L 193.189883 77.840169 \n",
       "L 193.703642 78.628417 \n",
       "L 194.214726 79.438875 \n",
       "L 194.723117 80.271393 \n",
       "L 195.228798 81.125822 \n",
       "L 195.731752 82.00201 \n",
       "L 196.231963 82.899803 \n",
       "L 196.729416 83.819047 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 148.54584 107.28887 \n",
       "L 149.193253 105.684526 \n",
       "L 149.840797 104.105056 \n",
       "L 150.488431 102.550581 \n",
       "L 151.136115 101.02122 \n",
       "L 151.783807 99.517086 \n",
       "L 152.431468 98.038294 \n",
       "L 153.079057 96.584951 \n",
       "L 153.726533 95.157164 \n",
       "L 154.373856 93.755037 \n",
       "L 155.020985 92.378668 \n",
       "L 155.667879 91.028155 \n",
       "L 156.314498 89.703591 \n",
       "L 156.9608 88.405067 \n",
       "L 157.606746 87.132668 \n",
       "L 158.252295 85.886479 \n",
       "L 158.897407 84.666581 \n",
       "L 159.54204 83.473051 \n",
       "L 160.186154 82.305962 \n",
       "L 160.82971 81.165386 \n",
       "L 161.472666 80.051388 \n",
       "L 162.114983 78.964034 \n",
       "L 162.756621 77.903383 \n",
       "L 163.397539 76.869494 \n",
       "L 164.037698 75.862419 \n",
       "L 164.677058 74.882208 \n",
       "L 165.315579 73.92891 \n",
       "L 165.953222 73.002567 \n",
       "L 166.589947 72.10322 \n",
       "L 167.225716 71.230905 \n",
       "L 167.860488 70.385655 \n",
       "L 168.494227 69.567501 \n",
       "L 169.126892 68.776469 \n",
       "L 169.758445 68.012582 \n",
       "L 170.388848 67.275859 \n",
       "L 171.018062 66.566317 \n",
       "L 171.646051 65.883968 \n",
       "L 172.272775 65.228822 \n",
       "L 172.898198 64.600885 \n",
       "L 173.522282 64.000159 \n",
       "L 174.144991 63.426644 \n",
       "L 174.766287 62.880334 \n",
       "L 175.386135 62.361223 \n",
       "L 176.004497 61.8693 \n",
       "L 176.621337 61.404549 \n",
       "L 177.236621 60.966954 \n",
       "L 177.850312 60.556493 \n",
       "L 178.462376 60.173142 \n",
       "L 179.072777 59.816873 \n",
       "L 179.681481 59.487656 \n",
       "L 180.288453 59.185455 \n",
       "L 180.89366 58.910235 \n",
       "L 181.497068 58.661953 \n",
       "L 182.098643 58.440566 \n",
       "L 182.698353 58.246027 \n",
       "L 183.296165 58.078285 \n",
       "L 183.892047 57.937288 \n",
       "L 184.485966 57.822977 \n",
       "L 185.07789 57.735295 \n",
       "L 185.667789 57.674178 \n",
       "L 186.255632 57.63956 \n",
       "L 186.841388 57.631373 \n",
       "L 187.425026 57.649544 \n",
       "L 188.006517 57.694 \n",
       "L 188.585832 57.764661 \n",
       "L 189.16294 57.861448 \n",
       "L 189.737815 57.984277 \n",
       "L 190.310426 58.133062 \n",
       "L 190.880747 58.307713 \n",
       "L 191.448749 58.508139 \n",
       "L 192.014406 58.734245 \n",
       "L 192.577691 58.985933 \n",
       "L 193.138578 59.263104 \n",
       "L 193.697039 59.565654 \n",
       "L 194.253051 59.893479 \n",
       "L 194.806588 60.24647 \n",
       "L 195.357625 60.624517 \n",
       "L 195.906138 61.027507 \n",
       "L 196.452103 61.455325 \n",
       "L 196.995497 61.907853 \n",
       "L 197.536297 62.38497 \n",
       "L 198.074479 62.886554 \n",
       "L 198.610023 63.41248 \n",
       "L 199.142906 63.962621 \n",
       "L 199.673106 64.536848 \n",
       "L 200.200604 65.135029 \n",
       "L 200.725379 65.75703 \n",
       "L 201.24741 66.402715 \n",
       "L 201.766679 67.071948 \n",
       "L 202.283166 67.764587 \n",
       "L 202.796852 68.48049 \n",
       "L 203.307719 69.219515 \n",
       "L 203.81575 69.981516 \n",
       "L 204.320927 70.766344 \n",
       "L 204.823234 71.57385 \n",
       "L 205.322653 72.403884 \n",
       "L 205.819169 73.256293 \n",
       "L 206.312766 74.130921 \n",
       "L 206.803429 75.027613 \n",
       "L 207.291143 75.946212 \n",
       "L 207.775894 76.886556 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 160.008622 97.415821 \n",
       "L 160.668624 95.78632 \n",
       "L 161.328524 94.182226 \n",
       "L 161.988279 92.603665 \n",
       "L 162.647848 91.050757 \n",
       "L 163.307188 89.523623 \n",
       "L 163.966258 88.022377 \n",
       "L 164.625015 86.547133 \n",
       "L 165.283417 85.097999 \n",
       "L 165.941423 83.675082 \n",
       "L 166.59899 82.278484 \n",
       "L 167.256077 80.908305 \n",
       "L 167.912641 79.564641 \n",
       "L 168.56864 78.247585 \n",
       "L 169.224033 76.957226 \n",
       "L 169.878778 75.69365 \n",
       "L 170.532834 74.456941 \n",
       "L 171.186158 73.247178 \n",
       "L 171.838709 72.064435 \n",
       "L 172.490446 70.908787 \n",
       "L 173.141328 69.780302 \n",
       "L 173.791312 68.679045 \n",
       "L 174.440359 67.605079 \n",
       "L 175.088427 66.558461 \n",
       "L 175.735476 65.539248 \n",
       "L 176.381465 64.547491 \n",
       "L 177.026353 63.583237 \n",
       "L 177.670101 62.646531 \n",
       "L 178.312667 61.737414 \n",
       "L 178.954013 60.855924 \n",
       "L 179.594098 60.002094 \n",
       "L 180.232883 59.175954 \n",
       "L 180.870329 58.377532 \n",
       "L 181.506396 57.606851 \n",
       "L 182.141047 56.86393 \n",
       "L 182.774241 56.148785 \n",
       "L 183.405941 55.461429 \n",
       "L 184.036109 54.801871 \n",
       "L 184.664706 54.170117 \n",
       "L 185.291696 53.566169 \n",
       "L 185.91704 52.990024 \n",
       "L 186.540702 52.441678 \n",
       "L 187.162646 51.921123 \n",
       "L 187.782833 51.428346 \n",
       "L 188.40123 50.963332 \n",
       "L 189.017799 50.526062 \n",
       "L 189.632504 50.116513 \n",
       "L 190.245312 49.73466 \n",
       "L 190.856187 49.380474 \n",
       "L 191.465093 49.053921 \n",
       "L 192.071999 48.754966 \n",
       "L 192.676868 48.483569 \n",
       "L 193.279668 48.239687 \n",
       "L 193.880366 48.023275 \n",
       "L 194.478928 47.834283 \n",
       "L 195.075323 47.672658 \n",
       "L 195.669518 47.538345 \n",
       "L 196.261482 47.431284 \n",
       "L 196.851183 47.351413 \n",
       "L 197.438591 47.298666 \n",
       "L 198.023674 47.272975 \n",
       "L 198.606404 47.274268 \n",
       "L 199.186749 47.30247 \n",
       "L 199.764682 47.357503 \n",
       "L 200.340172 47.439286 \n",
       "L 200.913192 47.547735 \n",
       "L 201.483713 47.682763 \n",
       "L 202.051708 47.84428 \n",
       "L 202.617149 48.032194 \n",
       "L 203.180011 48.246408 \n",
       "L 203.740265 48.486825 \n",
       "L 204.297888 48.753342 \n",
       "L 204.852852 49.045857 \n",
       "L 205.405133 49.364261 \n",
       "L 205.954707 49.708446 \n",
       "L 206.50155 50.078299 \n",
       "L 207.045637 50.473706 \n",
       "L 207.586945 50.894549 \n",
       "L 208.125452 51.340709 \n",
       "L 208.661135 51.812063 \n",
       "L 209.193973 52.308487 \n",
       "L 209.723944 52.829852 \n",
       "L 210.251026 53.37603 \n",
       "L 210.775201 53.946889 \n",
       "L 211.296447 54.542294 \n",
       "L 211.814745 55.162109 \n",
       "L 212.330076 55.806196 \n",
       "L 212.842421 56.474413 \n",
       "L 213.351762 57.166618 \n",
       "L 213.858081 57.882665 \n",
       "L 214.361362 58.622408 \n",
       "L 214.861587 59.385698 \n",
       "L 215.35874 60.172384 \n",
       "L 215.852805 60.982312 \n",
       "L 216.343767 61.815329 \n",
       "L 216.831611 62.671277 \n",
       "L 217.316323 63.549999 \n",
       "L 217.797888 64.451334 \n",
       "L 218.276292 65.375121 \n",
       "L 218.751524 66.321197 \n",
       "L 219.22357 67.289397 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 50.633789 79.513765 \n",
       "L 51.769057 80.956581 \n",
       "L 52.901267 82.372049 \n",
       "L 54.030491 83.760302 \n",
       "L 55.156804 85.121471 \n",
       "L 56.280278 86.455683 \n",
       "L 57.400987 87.763061 \n",
       "L 58.519002 89.043725 \n",
       "L 59.634395 90.29779 \n",
       "L 60.747237 91.52537 \n",
       "L 61.857599 92.726574 \n",
       "L 62.965551 93.901507 \n",
       "L 64.071164 95.050273 \n",
       "L 65.174507 96.172971 \n",
       "L 66.275648 97.269696 \n",
       "L 67.374657 98.340542 \n",
       "L 68.471603 99.385598 \n",
       "L 69.566553 100.40495 \n",
       "L 70.659575 101.398681 \n",
       "L 71.750737 102.366872 \n",
       "L 72.840106 103.309598 \n",
       "L 73.927748 104.226933 \n",
       "L 75.013732 105.118947 \n",
       "L 76.098122 105.985709 \n",
       "L 77.180985 106.827282 \n",
       "L 78.262387 107.643727 \n",
       "L 79.342394 108.435103 \n",
       "L 80.421072 109.201464 \n",
       "L 81.498485 109.942863 \n",
       "L 82.574699 110.659347 \n",
       "L 83.649779 111.350964 \n",
       "L 84.72379 112.017757 \n",
       "L 85.796797 112.659764 \n",
       "L 86.868864 113.277024 \n",
       "L 87.940056 113.86957 \n",
       "L 89.010437 114.437432 \n",
       "L 90.080072 114.980641 \n",
       "L 91.149025 115.499219 \n",
       "L 92.21736 115.993189 \n",
       "L 93.285142 116.462571 \n",
       "L 94.352434 116.907381 \n",
       "L 95.419301 117.327632 \n",
       "L 96.485807 117.723333 \n",
       "L 97.552016 118.094494 \n",
       "L 98.617993 118.441117 \n",
       "L 99.683802 118.763204 \n",
       "L 100.749506 119.060754 \n",
       "L 101.815171 119.333762 \n",
       "L 102.88086 119.582221 \n",
       "L 103.946639 119.80612 \n",
       "L 105.012571 120.005446 \n",
       "L 106.078723 120.180182 \n",
       "L 107.145157 120.330309 \n",
       "L 108.211941 120.455803 \n",
       "L 109.279138 120.556641 \n",
       "L 110.346814 120.632792 \n",
       "L 111.415035 120.684227 \n",
       "L 112.483866 120.710908 \n",
       "L 113.553374 120.7128 \n",
       "L 114.623624 120.689861 \n",
       "L 115.694682 120.642048 \n",
       "L 116.766617 120.569312 \n",
       "L 117.839493 120.471604 \n",
       "L 118.91338 120.348871 \n",
       "L 119.988343 120.201055 \n",
       "L 121.064452 120.028097 \n",
       "L 122.141773 119.829934 \n",
       "L 123.220376 119.606499 \n",
       "L 124.300329 119.357724 \n",
       "L 125.381702 119.083534 \n",
       "L 126.464565 118.783854 \n",
       "L 127.548986 118.458604 \n",
       "L 128.635037 118.107701 \n",
       "L 129.722789 117.731058 \n",
       "L 130.812312 117.328586 \n",
       "L 131.903678 116.900191 \n",
       "L 132.996961 116.445777 \n",
       "L 134.092232 115.965242 \n",
       "L 135.189565 115.458483 \n",
       "L 136.289033 114.925391 \n",
       "L 137.390712 114.365856 \n",
       "L 138.494675 113.779762 \n",
       "L 139.600999 113.166991 \n",
       "L 140.70976 112.527419 \n",
       "L 141.821033 111.86092 \n",
       "L 142.934898 111.167363 \n",
       "L 144.051431 110.446614 \n",
       "L 145.170712 109.698535 \n",
       "L 146.292821 108.922983 \n",
       "L 147.417836 108.119812 \n",
       "L 148.54584 107.28887 \n",
       "L 149.676913 106.430003 \n",
       "L 150.81114 105.543051 \n",
       "L 151.948602 104.627851 \n",
       "L 153.089384 103.684235 \n",
       "L 154.233571 102.71203 \n",
       "L 155.381249 101.71106 \n",
       "L 156.532505 100.681143 \n",
       "L 157.687427 99.622092 \n",
       "L 158.846102 98.533717 \n",
       "L 160.008622 97.415821 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 56.71769 64.942641 \n",
       "L 57.851789 66.397486 \n",
       "L 58.982946 67.824613 \n",
       "L 60.111237 69.224157 \n",
       "L 61.236734 70.596249 \n",
       "L 62.359511 71.941018 \n",
       "L 63.479639 73.258588 \n",
       "L 64.597192 74.549081 \n",
       "L 65.71224 75.812614 \n",
       "L 66.824856 77.049302 \n",
       "L 67.935109 78.259255 \n",
       "L 69.04307 79.442579 \n",
       "L 70.148809 80.59938 \n",
       "L 71.252395 81.729756 \n",
       "L 72.353899 82.833806 \n",
       "L 73.453388 83.911623 \n",
       "L 74.550931 84.963298 \n",
       "L 75.646596 85.988917 \n",
       "L 76.740452 86.988565 \n",
       "L 77.832566 87.962322 \n",
       "L 78.923004 88.910265 \n",
       "L 80.011835 89.832469 \n",
       "L 81.099126 90.729005 \n",
       "L 82.184942 91.599941 \n",
       "L 83.269349 92.445342 \n",
       "L 84.352416 93.265269 \n",
       "L 85.434206 94.05978 \n",
       "L 86.514786 94.828932 \n",
       "L 87.594221 95.572776 \n",
       "L 88.672578 96.291362 \n",
       "L 89.749921 96.984736 \n",
       "L 90.826315 97.652941 \n",
       "L 91.901826 98.296017 \n",
       "L 92.976518 98.914002 \n",
       "L 94.050456 99.50693 \n",
       "L 95.123706 100.07483 \n",
       "L 96.196331 100.617732 \n",
       "L 97.268397 101.135661 \n",
       "L 98.339968 101.628638 \n",
       "L 99.411109 102.096683 \n",
       "L 100.481884 102.539811 \n",
       "L 101.552358 102.958035 \n",
       "L 102.622595 103.351365 \n",
       "L 103.692661 103.719809 \n",
       "L 104.762619 104.06337 \n",
       "L 105.832535 104.38205 \n",
       "L 106.902473 104.675845 \n",
       "L 107.972499 104.944752 \n",
       "L 109.042676 105.188761 \n",
       "L 110.113071 105.407862 \n",
       "L 111.183749 105.602041 \n",
       "L 112.254774 105.77128 \n",
       "L 113.326213 105.915559 \n",
       "L 114.39813 106.034855 \n",
       "L 115.470593 106.129141 \n",
       "L 116.543666 106.198388 \n",
       "L 117.617416 106.242563 \n",
       "L 118.691909 106.26163 \n",
       "L 119.767212 106.25555 \n",
       "L 120.843392 106.224283 \n",
       "L 121.920516 106.167781 \n",
       "L 122.998652 106.085997 \n",
       "L 124.077866 105.978879 \n",
       "L 125.158228 105.846373 \n",
       "L 126.239805 105.68842 \n",
       "L 127.322667 105.504959 \n",
       "L 128.406881 105.295926 \n",
       "L 129.492518 105.061252 \n",
       "L 130.579647 104.800866 \n",
       "L 131.668338 104.514694 \n",
       "L 132.758662 104.202657 \n",
       "L 133.85069 103.864675 \n",
       "L 134.944493 103.500661 \n",
       "L 136.040143 103.110528 \n",
       "L 137.137712 102.694183 \n",
       "L 138.237274 102.251532 \n",
       "L 139.338901 101.782475 \n",
       "L 140.442667 101.286909 \n",
       "L 141.548647 100.764728 \n",
       "L 142.656916 100.215822 \n",
       "L 143.76755 99.640077 \n",
       "L 144.880623 99.037376 \n",
       "L 145.996214 98.407596 \n",
       "L 147.114399 97.750613 \n",
       "L 148.235257 97.066297 \n",
       "L 149.358866 96.354515 \n",
       "L 150.485306 95.61513 \n",
       "L 151.614656 94.848001 \n",
       "L 152.746998 94.052981 \n",
       "L 153.882414 93.229921 \n",
       "L 155.020985 92.378668 \n",
       "L 156.162795 91.499064 \n",
       "L 157.307928 90.590945 \n",
       "L 158.456469 89.654144 \n",
       "L 159.608503 88.688491 \n",
       "L 160.764118 87.693809 \n",
       "L 161.9234 86.669918 \n",
       "L 163.086438 85.616633 \n",
       "L 164.253322 84.533763 \n",
       "L 165.424142 83.421113 \n",
       "L 166.59899 82.278484 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 62.977777 52.941634 \n",
       "L 64.108291 54.404023 \n",
       "L 65.235994 55.838435 \n",
       "L 66.360961 57.245007 \n",
       "L 67.483265 58.623872 \n",
       "L 68.602977 59.975159 \n",
       "L 69.720171 61.298993 \n",
       "L 70.834917 62.595495 \n",
       "L 71.947288 63.864786 \n",
       "L 73.057354 65.106978 \n",
       "L 74.165186 66.322185 \n",
       "L 75.270853 67.510512 \n",
       "L 76.374426 68.672066 \n",
       "L 77.475973 69.806946 \n",
       "L 78.575564 70.915251 \n",
       "L 79.673267 71.997074 \n",
       "L 80.76915 73.052507 \n",
       "L 81.863282 74.081637 \n",
       "L 82.95573 75.084548 \n",
       "L 84.046562 76.061322 \n",
       "L 85.135845 77.012037 \n",
       "L 86.223645 77.936766 \n",
       "L 87.31003 78.835582 \n",
       "L 88.395066 79.708553 \n",
       "L 89.478819 80.555742 \n",
       "L 90.561355 81.377214 \n",
       "L 91.642741 82.173025 \n",
       "L 92.723041 82.943232 \n",
       "L 93.802322 83.687888 \n",
       "L 94.880649 84.407041 \n",
       "L 95.958087 85.100738 \n",
       "L 97.034701 85.769022 \n",
       "L 98.110557 86.411933 \n",
       "L 99.185719 87.029508 \n",
       "L 100.260252 87.621782 \n",
       "L 101.334222 88.188784 \n",
       "L 102.407693 88.730544 \n",
       "L 103.480729 89.247086 \n",
       "L 104.553396 89.738432 \n",
       "L 105.625759 90.2046 \n",
       "L 106.697881 90.645606 \n",
       "L 107.769829 91.061464 \n",
       "L 108.841665 91.452182 \n",
       "L 109.913457 91.817767 \n",
       "L 110.985267 92.158224 \n",
       "L 112.057162 92.473551 \n",
       "L 113.129206 92.763748 \n",
       "L 114.201464 93.028808 \n",
       "L 115.274003 93.268723 \n",
       "L 116.346886 93.483482 \n",
       "L 117.42018 93.673068 \n",
       "L 118.493949 93.837466 \n",
       "L 119.568261 93.976653 \n",
       "L 120.643181 94.090605 \n",
       "L 121.718775 94.179297 \n",
       "L 122.79511 94.242697 \n",
       "L 123.872252 94.280772 \n",
       "L 124.950268 94.293485 \n",
       "L 126.029224 94.280798 \n",
       "L 127.109189 94.242666 \n",
       "L 128.190231 94.179044 \n",
       "L 129.272416 94.089883 \n",
       "L 130.355813 93.97513 \n",
       "L 131.440491 93.834729 \n",
       "L 132.526518 93.668622 \n",
       "L 133.613964 93.476746 \n",
       "L 134.702898 93.259034 \n",
       "L 135.79339 93.015419 \n",
       "L 136.885511 92.745828 \n",
       "L 137.979332 92.450184 \n",
       "L 139.074922 92.128409 \n",
       "L 140.172355 91.78042 \n",
       "L 141.271702 91.40613 \n",
       "L 142.373036 91.005451 \n",
       "L 143.47643 90.578288 \n",
       "L 144.581957 90.124544 \n",
       "L 145.689691 89.64412 \n",
       "L 146.799708 89.13691 \n",
       "L 147.912082 88.602808 \n",
       "L 149.026889 88.041701 \n",
       "L 150.144205 87.453473 \n",
       "L 151.264108 86.838006 \n",
       "L 152.386676 86.195177 \n",
       "L 153.511985 85.524857 \n",
       "L 154.640116 84.826917 \n",
       "L 155.771148 84.101221 \n",
       "L 156.905162 83.34763 \n",
       "L 158.042238 82.566001 \n",
       "L 159.182459 81.756186 \n",
       "L 160.325907 80.918034 \n",
       "L 161.472666 80.051388 \n",
       "L 162.62282 79.156089 \n",
       "L 163.776454 78.231971 \n",
       "L 164.933654 77.278866 \n",
       "L 166.094507 76.2966 \n",
       "L 167.259101 75.284994 \n",
       "L 168.427525 74.243866 \n",
       "L 169.599868 73.173027 \n",
       "L 170.776222 72.072286 \n",
       "L 171.956677 70.941445 \n",
       "L 173.141328 69.780302 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 69.376017 43.585267 \n",
       "L 70.500575 45.050619 \n",
       "L 71.622463 46.487853 \n",
       "L 72.741757 47.897106 \n",
       "L 73.858527 49.278509 \n",
       "L 74.972846 50.632194 \n",
       "L 76.084785 51.958284 \n",
       "L 77.194416 53.256903 \n",
       "L 78.301809 54.528169 \n",
       "L 79.407034 55.772196 \n",
       "L 80.510161 56.989096 \n",
       "L 81.61126 58.178977 \n",
       "L 82.710399 59.341943 \n",
       "L 83.807648 60.478096 \n",
       "L 84.903075 61.587533 \n",
       "L 85.996747 62.670348 \n",
       "L 87.088734 63.726632 \n",
       "L 88.179102 64.756474 \n",
       "L 89.267918 65.759957 \n",
       "L 90.35525 66.737162 \n",
       "L 91.441164 67.688167 \n",
       "L 92.525727 68.613047 \n",
       "L 93.609006 69.511873 \n",
       "L 94.691065 70.384713 \n",
       "L 95.771972 71.231632 \n",
       "L 96.851791 72.052692 \n",
       "L 97.930589 72.847952 \n",
       "L 99.008431 73.617467 \n",
       "L 100.085382 74.361289 \n",
       "L 101.161507 75.079468 \n",
       "L 102.236871 75.772049 \n",
       "L 103.311539 76.439077 \n",
       "L 104.385577 77.08059 \n",
       "L 105.459048 77.696627 \n",
       "L 106.532018 78.287219 \n",
       "L 107.604551 78.852399 \n",
       "L 108.676712 79.392195 \n",
       "L 109.748566 79.90663 \n",
       "L 110.820176 80.395726 \n",
       "L 111.891608 80.859502 \n",
       "L 112.962927 81.297973 \n",
       "L 114.034196 81.711153 \n",
       "L 115.105481 82.099049 \n",
       "L 116.176847 82.461669 \n",
       "L 117.248357 82.799016 \n",
       "L 118.320078 83.11109 \n",
       "L 119.392075 83.397888 \n",
       "L 120.464411 83.659405 \n",
       "L 121.537153 83.895631 \n",
       "L 122.610367 84.106555 \n",
       "L 123.684116 84.292161 \n",
       "L 124.758468 84.452431 \n",
       "L 125.833488 84.587345 \n",
       "L 126.909242 84.696877 \n",
       "L 127.985797 84.780999 \n",
       "L 129.063218 84.839683 \n",
       "L 130.141574 84.872892 \n",
       "L 131.22093 84.880592 \n",
       "L 132.301353 84.862741 \n",
       "L 133.382913 84.819296 \n",
       "L 134.465675 84.75021 \n",
       "L 135.549709 84.655434 \n",
       "L 136.635082 84.534915 \n",
       "L 137.721864 84.388596 \n",
       "L 138.810123 84.216418 \n",
       "L 139.89993 84.018317 \n",
       "L 140.991353 83.794229 \n",
       "L 142.084464 83.544082 \n",
       "L 143.179332 83.267803 \n",
       "L 144.27603 82.965318 \n",
       "L 145.374628 82.636544 \n",
       "L 146.475198 82.2814 \n",
       "L 147.577813 81.899798 \n",
       "L 148.682546 81.491648 \n",
       "L 149.78947 81.056856 \n",
       "L 150.89866 80.595323 \n",
       "L 152.01019 80.106949 \n",
       "L 153.124134 79.59163 \n",
       "L 154.240569 79.049255 \n",
       "L 155.359572 78.479713 \n",
       "L 156.481218 77.882888 \n",
       "L 157.605585 77.258659 \n",
       "L 158.732752 76.606903 \n",
       "L 159.862797 75.92749 \n",
       "L 160.995801 75.220291 \n",
       "L 162.131843 74.485168 \n",
       "L 163.271004 73.721982 \n",
       "L 164.413366 72.930588 \n",
       "L 165.559012 72.110838 \n",
       "L 166.708024 71.262579 \n",
       "L 167.860488 70.385655 \n",
       "L 169.016489 69.479905 \n",
       "L 170.176111 68.545162 \n",
       "L 171.339442 67.581256 \n",
       "L 172.50657 66.588013 \n",
       "L 173.677583 65.565253 \n",
       "L 174.852571 64.512792 \n",
       "L 176.031624 63.430442 \n",
       "L 177.214835 62.318009 \n",
       "L 178.402295 61.175294 \n",
       "L 179.594098 60.002094 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 75.872827 36.915722 \n",
       "L 76.989129 38.37944 \n",
       "L 78.102914 39.815015 \n",
       "L 79.214253 41.222583 \n",
       "L 80.323219 42.602277 \n",
       "L 81.429881 43.954225 \n",
       "L 82.534311 45.278554 \n",
       "L 83.636578 46.575383 \n",
       "L 84.736753 47.844831 \n",
       "L 85.834904 49.087012 \n",
       "L 86.931101 50.302038 \n",
       "L 88.025412 51.490015 \n",
       "L 89.117906 52.651048 \n",
       "L 90.20865 53.785238 \n",
       "L 91.297711 54.892681 \n",
       "L 92.385159 55.973471 \n",
       "L 93.471058 57.027698 \n",
       "L 94.555477 58.055451 \n",
       "L 95.638481 59.056813 \n",
       "L 96.720138 60.031864 \n",
       "L 97.800512 60.980683 \n",
       "L 98.87967 61.903342 \n",
       "L 99.957678 62.799913 \n",
       "L 101.0346 63.670465 \n",
       "L 102.110503 64.51506 \n",
       "L 103.185451 65.333761 \n",
       "L 104.25951 66.126627 \n",
       "L 105.332743 66.893712 \n",
       "L 106.405217 67.635068 \n",
       "L 107.476995 68.350744 \n",
       "L 108.548142 69.040787 \n",
       "L 109.618722 69.705238 \n",
       "L 110.6888 70.344137 \n",
       "L 111.75844 70.957522 \n",
       "L 112.827706 71.545425 \n",
       "L 113.896663 72.107878 \n",
       "L 114.965375 72.644907 \n",
       "L 116.033905 73.156536 \n",
       "L 117.102319 73.642788 \n",
       "L 118.17068 74.103681 \n",
       "L 119.239052 74.539229 \n",
       "L 120.307501 74.949445 \n",
       "L 121.37609 75.334338 \n",
       "L 122.444883 75.693914 \n",
       "L 123.513946 76.028177 \n",
       "L 124.583343 76.337126 \n",
       "L 125.653138 76.620759 \n",
       "L 126.723397 76.879068 \n",
       "L 127.794184 77.112046 \n",
       "L 128.865565 77.31968 \n",
       "L 129.937605 77.501954 \n",
       "L 131.010369 77.658851 \n",
       "L 132.083923 77.790348 \n",
       "L 133.158333 77.896422 \n",
       "L 134.233665 77.977044 \n",
       "L 135.309986 78.032184 \n",
       "L 136.387362 78.061808 \n",
       "L 137.465859 78.065878 \n",
       "L 138.545546 78.044355 \n",
       "L 139.626489 77.997195 \n",
       "L 140.708756 77.924351 \n",
       "L 141.792415 77.825773 \n",
       "L 142.877534 77.701409 \n",
       "L 143.964183 77.551201 \n",
       "L 145.052429 77.375091 \n",
       "L 146.142344 77.173014 \n",
       "L 147.233996 76.944905 \n",
       "L 148.327456 76.690694 \n",
       "L 149.422794 76.410308 \n",
       "L 150.520082 76.10367 \n",
       "L 151.619391 75.770701 \n",
       "L 152.720793 75.411317 \n",
       "L 153.82436 75.025431 \n",
       "L 154.930167 74.612952 \n",
       "L 156.038286 74.173787 \n",
       "L 157.148791 73.707838 \n",
       "L 158.261757 73.215003 \n",
       "L 159.37726 72.695178 \n",
       "L 160.495374 72.148254 \n",
       "L 161.616178 71.574119 \n",
       "L 162.739746 70.972657 \n",
       "L 163.866159 70.343746 \n",
       "L 164.995493 69.687264 \n",
       "L 166.127827 69.003083 \n",
       "L 167.263243 68.291071 \n",
       "L 168.401819 67.551091 \n",
       "L 169.543638 66.783004 \n",
       "L 170.688782 65.986667 \n",
       "L 171.837332 65.16193 \n",
       "L 172.989374 64.308641 \n",
       "L 174.144991 63.426644 \n",
       "L 175.304269 62.515777 \n",
       "L 176.467294 61.575875 \n",
       "L 177.634153 60.606767 \n",
       "L 178.804934 59.608281 \n",
       "L 179.979726 58.580235 \n",
       "L 181.158619 57.522447 \n",
       "L 182.341705 56.434727 \n",
       "L 183.529075 55.316883 \n",
       "L 184.720822 54.168716 \n",
       "L 185.91704 52.990024 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 82.427882 32.94229 \n",
       "L 83.53373 34.399832 \n",
       "L 84.63722 35.829323 \n",
       "L 85.738422 37.230898 \n",
       "L 86.837406 38.604688 \n",
       "L 87.934241 39.95082 \n",
       "L 89.028997 41.269418 \n",
       "L 90.121742 42.560603 \n",
       "L 91.212546 43.824491 \n",
       "L 92.301476 45.061196 \n",
       "L 93.3886 46.270827 \n",
       "L 94.473986 47.453491 \n",
       "L 95.557701 48.609291 \n",
       "L 96.639812 49.738326 \n",
       "L 97.720384 50.840693 \n",
       "L 98.799485 51.916484 \n",
       "L 99.877181 52.96579 \n",
       "L 100.953537 53.988696 \n",
       "L 102.028619 54.985285 \n",
       "L 103.102492 55.955638 \n",
       "L 104.175221 56.899831 \n",
       "L 105.246871 57.817937 \n",
       "L 106.317507 58.710027 \n",
       "L 107.387194 59.576167 \n",
       "L 108.455995 60.416422 \n",
       "L 109.523976 61.230852 \n",
       "L 110.5912 62.019515 \n",
       "L 111.657731 62.782466 \n",
       "L 112.723633 63.519756 \n",
       "L 113.788971 64.231434 \n",
       "L 114.853807 64.917544 \n",
       "L 115.918205 65.578129 \n",
       "L 116.982229 66.213228 \n",
       "L 118.045943 66.822878 \n",
       "L 119.10941 67.407112 \n",
       "L 120.172694 67.965958 \n",
       "L 121.235858 68.499446 \n",
       "L 122.298965 69.007598 \n",
       "L 123.36208 69.490436 \n",
       "L 124.425265 69.947978 \n",
       "L 125.488586 70.380239 \n",
       "L 126.552104 70.78723 \n",
       "L 127.615884 71.168961 \n",
       "L 128.67999 71.525437 \n",
       "L 129.744486 71.856662 \n",
       "L 130.809436 72.162636 \n",
       "L 131.874903 72.443354 \n",
       "L 132.940953 72.698812 \n",
       "L 134.00765 72.929 \n",
       "L 135.075059 73.133906 \n",
       "L 136.143243 73.313514 \n",
       "L 137.212269 73.467806 \n",
       "L 138.282202 73.59676 \n",
       "L 139.353106 73.700353 \n",
       "L 140.425048 73.778557 \n",
       "L 141.498094 73.83134 \n",
       "L 142.572309 73.858669 \n",
       "L 143.647761 73.860508 \n",
       "L 144.724515 73.836816 \n",
       "L 145.802639 73.787549 \n",
       "L 146.882201 73.712662 \n",
       "L 147.963267 73.612104 \n",
       "L 149.045906 73.485824 \n",
       "L 150.130187 73.333764 \n",
       "L 151.216177 73.155866 \n",
       "L 152.303947 72.952067 \n",
       "L 153.393565 72.7223 \n",
       "L 154.485102 72.466497 \n",
       "L 155.578627 72.184586 \n",
       "L 156.674213 71.876489 \n",
       "L 157.77193 71.542128 \n",
       "L 158.871849 71.181419 \n",
       "L 159.974044 70.794277 \n",
       "L 161.078587 70.380612 \n",
       "L 162.185551 69.94033 \n",
       "L 163.29501 69.473334 \n",
       "L 164.407039 68.979523 \n",
       "L 165.521712 68.458795 \n",
       "L 166.639106 67.91104 \n",
       "L 167.759297 67.336147 \n",
       "L 168.882361 66.734001 \n",
       "L 170.008375 66.104482 \n",
       "L 171.13742 65.447468 \n",
       "L 172.269572 64.762832 \n",
       "L 173.404912 64.050442 \n",
       "L 174.54352 63.310165 \n",
       "L 175.685478 62.541861 \n",
       "L 176.830867 61.745387 \n",
       "L 177.97977 60.920596 \n",
       "L 179.132271 60.067338 \n",
       "L 180.288453 59.185455 \n",
       "L 181.448403 58.27479 \n",
       "L 182.612206 57.335177 \n",
       "L 183.779949 56.366447 \n",
       "L 184.951721 55.368429 \n",
       "L 186.12761 54.340943 \n",
       "L 187.307706 53.283808 \n",
       "L 188.4921 52.196837 \n",
       "L 189.680885 51.079838 \n",
       "L 190.874153 49.932615 \n",
       "L 192.071999 48.754966 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 89.000935 31.641811 \n",
       "L 90.094257 33.088766 \n",
       "L 91.185384 34.507874 \n",
       "L 92.274385 35.89927 \n",
       "L 93.361328 37.26308 \n",
       "L 94.446282 38.599433 \n",
       "L 95.529314 39.908448 \n",
       "L 96.610492 41.190245 \n",
       "L 97.689883 42.444939 \n",
       "L 98.767554 43.672642 \n",
       "L 99.843571 44.873461 \n",
       "L 100.918 46.047501 \n",
       "L 101.990907 47.194864 \n",
       "L 103.062358 48.315648 \n",
       "L 104.132417 49.409948 \n",
       "L 105.20115 50.477854 \n",
       "L 106.268621 51.519457 \n",
       "L 107.334896 52.53484 \n",
       "L 108.400037 53.524085 \n",
       "L 109.464111 54.487272 \n",
       "L 110.527179 55.424474 \n",
       "L 111.589307 56.335766 \n",
       "L 112.650558 57.221216 \n",
       "L 113.710994 58.080889 \n",
       "L 114.770681 58.91485 \n",
       "L 115.829679 59.723157 \n",
       "L 116.888054 60.505868 \n",
       "L 117.945867 61.263037 \n",
       "L 119.003181 61.994714 \n",
       "L 120.06006 62.700946 \n",
       "L 121.116566 63.381779 \n",
       "L 122.172761 64.037253 \n",
       "L 123.228709 64.667409 \n",
       "L 124.284471 65.27228 \n",
       "L 125.340111 65.851901 \n",
       "L 126.395692 66.406299 \n",
       "L 127.451275 66.935503 \n",
       "L 128.506923 67.439535 \n",
       "L 129.562699 67.918417 \n",
       "L 130.618667 68.372166 \n",
       "L 131.674887 68.800797 \n",
       "L 132.731424 69.204321 \n",
       "L 133.78834 69.582748 \n",
       "L 134.845699 69.936082 \n",
       "L 135.903563 70.264328 \n",
       "L 136.961996 70.567484 \n",
       "L 138.021061 70.845548 \n",
       "L 139.080822 71.098514 \n",
       "L 140.141342 71.326372 \n",
       "L 141.202685 71.529111 \n",
       "L 142.264916 71.706715 \n",
       "L 143.328099 71.859165 \n",
       "L 144.392297 71.986442 \n",
       "L 145.457577 72.088521 \n",
       "L 146.524002 72.165374 \n",
       "L 147.591639 72.216971 \n",
       "L 148.660551 72.243279 \n",
       "L 149.730807 72.244261 \n",
       "L 150.80247 72.219878 \n",
       "L 151.875609 72.170087 \n",
       "L 152.950288 72.094842 \n",
       "L 154.026577 71.994095 \n",
       "L 155.104541 71.867793 \n",
       "L 156.184248 71.71588 \n",
       "L 157.265767 71.538299 \n",
       "L 158.349167 71.334987 \n",
       "L 159.434515 71.10588 \n",
       "L 160.521883 70.850909 \n",
       "L 161.611338 70.570002 \n",
       "L 162.702952 70.263085 \n",
       "L 163.796795 69.930079 \n",
       "L 164.892939 69.570903 \n",
       "L 165.991455 69.185471 \n",
       "L 167.092415 68.773694 \n",
       "L 168.195893 68.335482 \n",
       "L 169.301961 67.870737 \n",
       "L 170.410694 67.379362 \n",
       "L 171.522166 66.861253 \n",
       "L 172.636452 66.316304 \n",
       "L 173.753627 65.744405 \n",
       "L 174.873769 65.145442 \n",
       "L 175.996954 64.519298 \n",
       "L 177.123261 63.865851 \n",
       "L 178.252766 63.184978 \n",
       "L 179.38555 62.476547 \n",
       "L 180.521693 61.740428 \n",
       "L 181.661274 60.976482 \n",
       "L 182.804376 60.184569 \n",
       "L 183.951081 59.364545 \n",
       "L 185.101472 58.51626 \n",
       "L 186.255632 57.63956 \n",
       "L 187.413647 56.734289 \n",
       "L 188.575601 55.800285 \n",
       "L 189.741583 54.837381 \n",
       "L 190.911678 53.845408 \n",
       "L 192.085977 52.824189 \n",
       "L 193.264567 51.773546 \n",
       "L 194.44754 50.693294 \n",
       "L 195.634987 49.583245 \n",
       "L 196.827001 48.443205 \n",
       "L 198.023674 47.272975 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 95.552611 32.960061 \n",
       "L 96.631478 34.392215 \n",
       "L 97.708317 35.796834 \n",
       "L 98.783196 37.174051 \n",
       "L 99.85618 38.52399 \n",
       "L 100.927336 39.846775 \n",
       "L 101.996731 41.142527 \n",
       "L 103.06443 42.411359 \n",
       "L 104.130498 43.653387 \n",
       "L 105.195001 44.868718 \n",
       "L 106.258004 46.057459 \n",
       "L 107.31957 47.219712 \n",
       "L 108.379764 48.355578 \n",
       "L 109.438651 49.465152 \n",
       "L 110.496294 50.548527 \n",
       "L 111.552756 51.605792 \n",
       "L 112.6081 52.637036 \n",
       "L 113.66239 53.642339 \n",
       "L 114.715689 54.621784 \n",
       "L 115.768059 55.575447 \n",
       "L 116.819563 56.503402 \n",
       "L 117.870263 57.40572 \n",
       "L 118.920221 58.282468 \n",
       "L 119.9695 59.133712 \n",
       "L 121.018162 59.959514 \n",
       "L 122.066267 60.759931 \n",
       "L 123.113879 61.53502 \n",
       "L 124.161058 62.284833 \n",
       "L 125.207867 63.009421 \n",
       "L 126.254367 63.708829 \n",
       "L 127.300619 64.383102 \n",
       "L 128.346685 65.03228 \n",
       "L 129.392626 65.656401 \n",
       "L 130.438504 66.255501 \n",
       "L 131.48438 66.829611 \n",
       "L 132.530315 67.37876 \n",
       "L 133.576372 67.902974 \n",
       "L 134.622611 68.402277 \n",
       "L 135.669095 68.876689 \n",
       "L 136.715884 69.326227 \n",
       "L 137.763041 69.750906 \n",
       "L 138.810627 70.150737 \n",
       "L 139.858704 70.525729 \n",
       "L 140.907334 70.875888 \n",
       "L 141.956579 71.201216 \n",
       "L 143.006501 71.501713 \n",
       "L 144.057164 71.777376 \n",
       "L 145.108629 72.0282 \n",
       "L 146.160958 72.254175 \n",
       "L 147.214216 72.45529 \n",
       "L 148.268465 72.631529 \n",
       "L 149.323768 72.782876 \n",
       "L 150.380189 72.909308 \n",
       "L 151.437791 73.010804 \n",
       "L 152.49664 73.087335 \n",
       "L 153.556798 73.138872 \n",
       "L 154.61833 73.165383 \n",
       "L 155.681302 73.166832 \n",
       "L 156.745778 73.143179 \n",
       "L 157.811825 73.094385 \n",
       "L 158.879507 73.020402 \n",
       "L 159.94889 72.921184 \n",
       "L 161.020042 72.796679 \n",
       "L 162.093029 72.646834 \n",
       "L 163.167919 72.47159 \n",
       "L 164.244778 72.270888 \n",
       "L 165.323676 72.044663 \n",
       "L 166.40468 71.79285 \n",
       "L 167.487859 71.515376 \n",
       "L 168.573284 71.21217 \n",
       "L 169.661023 70.883154 \n",
       "L 170.751147 70.528249 \n",
       "L 171.843727 70.147371 \n",
       "L 172.938835 69.740433 \n",
       "L 174.036542 69.307346 \n",
       "L 175.13692 68.848015 \n",
       "L 176.240044 68.362343 \n",
       "L 177.345985 67.850231 \n",
       "L 178.45482 67.311573 \n",
       "L 179.566622 66.746262 \n",
       "L 180.681467 66.154188 \n",
       "L 181.799432 65.535234 \n",
       "L 182.920593 64.889282 \n",
       "L 184.045027 64.21621 \n",
       "L 185.172814 63.515891 \n",
       "L 186.304032 62.788196 \n",
       "L 187.43876 62.03299 \n",
       "L 188.57708 61.250136 \n",
       "L 189.719073 60.439492 \n",
       "L 190.864821 59.600911 \n",
       "L 192.014406 58.734245 \n",
       "L 193.167914 57.839338 \n",
       "L 194.325428 56.916033 \n",
       "L 195.487035 55.964167 \n",
       "L 196.65282 54.983574 \n",
       "L 197.822872 53.974081 \n",
       "L 198.997279 52.935513 \n",
       "L 200.176131 51.867691 \n",
       "L 201.359518 50.770429 \n",
       "L 202.547532 49.643539 \n",
       "L 203.740265 48.486825 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 102.045142 36.814003 \n",
       "L 103.107788 38.227396 \n",
       "L 104.168575 39.61367 \n",
       "L 105.227566 40.972951 \n",
       "L 106.284828 42.305361 \n",
       "L 107.340425 43.611022 \n",
       "L 108.394421 44.89005 \n",
       "L 109.446879 46.142558 \n",
       "L 110.497863 47.368655 \n",
       "L 111.547437 48.56845 \n",
       "L 112.595663 49.742044 \n",
       "L 113.642605 50.889538 \n",
       "L 114.688324 52.011029 \n",
       "L 115.732883 53.106612 \n",
       "L 116.776344 54.176376 \n",
       "L 117.818769 55.220409 \n",
       "L 118.860218 56.238796 \n",
       "L 119.900755 57.231617 \n",
       "L 120.94044 58.198952 \n",
       "L 121.979333 59.140875 \n",
       "L 123.017497 60.057458 \n",
       "L 124.054991 60.948772 \n",
       "L 125.091877 61.814881 \n",
       "L 126.128215 62.65585 \n",
       "L 127.164065 63.471738 \n",
       "L 128.199488 64.262603 \n",
       "L 129.234544 65.028499 \n",
       "L 130.269293 65.769477 \n",
       "L 131.303796 66.485587 \n",
       "L 132.338112 67.176874 \n",
       "L 133.372302 67.84338 \n",
       "L 134.406425 68.485146 \n",
       "L 135.440542 69.102208 \n",
       "L 136.474713 69.694601 \n",
       "L 137.508998 70.262356 \n",
       "L 138.543456 70.805501 \n",
       "L 139.578149 71.324062 \n",
       "L 140.613136 71.818061 \n",
       "L 141.648477 72.287519 \n",
       "L 142.684233 72.732453 \n",
       "L 143.720464 73.152876 \n",
       "L 144.757231 73.5488 \n",
       "L 145.794594 73.920234 \n",
       "L 146.832614 74.267183 \n",
       "L 147.871351 74.589651 \n",
       "L 148.910868 74.887636 \n",
       "L 149.951224 75.161137 \n",
       "L 150.992482 75.410147 \n",
       "L 152.034702 75.634657 \n",
       "L 153.077947 75.834658 \n",
       "L 154.122278 76.010133 \n",
       "L 155.167758 76.161067 \n",
       "L 156.214448 76.287438 \n",
       "L 157.262412 76.389225 \n",
       "L 158.311712 76.4664 \n",
       "L 159.362411 76.518935 \n",
       "L 160.414573 76.546799 \n",
       "L 161.468261 76.549956 \n",
       "L 162.523539 76.52837 \n",
       "L 163.580472 76.481999 \n",
       "L 164.639124 76.4108 \n",
       "L 165.69956 76.314726 \n",
       "L 166.761846 76.193727 \n",
       "L 167.826047 76.047751 \n",
       "L 168.892229 75.876743 \n",
       "L 169.960459 75.680642 \n",
       "L 171.030804 75.459388 \n",
       "L 172.103331 75.212915 \n",
       "L 173.178108 74.941155 \n",
       "L 174.255203 74.644037 \n",
       "L 175.334685 74.321485 \n",
       "L 176.416623 73.973423 \n",
       "L 177.501086 73.599768 \n",
       "L 178.588146 73.200437 \n",
       "L 179.677873 72.775342 \n",
       "L 180.770338 72.324392 \n",
       "L 181.865613 71.847493 \n",
       "L 182.96377 71.344546 \n",
       "L 184.064883 70.81545 \n",
       "L 185.169026 70.260101 \n",
       "L 186.276272 69.67839 \n",
       "L 187.386697 69.070206 \n",
       "L 188.500377 68.435433 \n",
       "L 189.617387 67.773951 \n",
       "L 190.737806 67.085639 \n",
       "L 191.86171 66.37037 \n",
       "L 192.989179 65.628013 \n",
       "L 194.120291 64.858434 \n",
       "L 195.255128 64.061496 \n",
       "L 196.393769 63.237057 \n",
       "L 197.536297 62.38497 \n",
       "L 198.682794 61.505086 \n",
       "L 199.833344 60.597252 \n",
       "L 200.988031 59.661308 \n",
       "L 202.146941 58.697093 \n",
       "L 203.310159 57.704441 \n",
       "L 204.477774 56.683181 \n",
       "L 205.649873 55.633137 \n",
       "L 206.826545 54.55413 \n",
       "L 208.007881 53.445976 \n",
       "L 209.193973 52.308487 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 108.443017 43.094743 \n",
       "L 109.487851 44.485725 \n",
       "L 110.530992 45.850093 \n",
       "L 111.572504 47.18797 \n",
       "L 112.61245 48.499477 \n",
       "L 113.650892 49.784729 \n",
       "L 114.687892 51.04384 \n",
       "L 115.723511 52.276919 \n",
       "L 116.757812 53.484074 \n",
       "L 117.790856 54.665407 \n",
       "L 118.822702 55.821019 \n",
       "L 119.853414 56.951008 \n",
       "L 120.88305 58.055467 \n",
       "L 121.911671 59.134487 \n",
       "L 122.939337 60.188158 \n",
       "L 123.966109 61.216562 \n",
       "L 124.992046 62.219784 \n",
       "L 126.017207 63.197901 \n",
       "L 127.041653 64.150989 \n",
       "L 128.065442 65.079122 \n",
       "L 129.088633 65.98237 \n",
       "L 130.111286 66.8608 \n",
       "L 131.133459 67.714476 \n",
       "L 132.155212 68.54346 \n",
       "L 133.176602 69.347811 \n",
       "L 134.19769 70.127583 \n",
       "L 135.218533 70.882831 \n",
       "L 136.23919 71.613604 \n",
       "L 137.259719 72.319949 \n",
       "L 138.280179 73.001911 \n",
       "L 139.300629 73.659531 \n",
       "L 140.321127 74.292848 \n",
       "L 141.341731 74.901898 \n",
       "L 142.3625 75.486715 \n",
       "L 143.383492 76.047329 \n",
       "L 144.404766 76.583767 \n",
       "L 145.426381 77.096055 \n",
       "L 146.448395 77.584215 \n",
       "L 147.470867 78.048266 \n",
       "L 148.493856 78.488225 \n",
       "L 149.517421 78.904106 \n",
       "L 150.54162 79.29592 \n",
       "L 151.566513 79.663675 \n",
       "L 152.592159 80.007377 \n",
       "L 153.618618 80.327029 \n",
       "L 154.645949 80.622631 \n",
       "L 155.674211 80.894181 \n",
       "L 156.703466 81.141673 \n",
       "L 157.733771 81.365099 \n",
       "L 158.765189 81.564448 \n",
       "L 159.79778 81.739707 \n",
       "L 160.831603 81.890858 \n",
       "L 161.866721 82.017883 \n",
       "L 162.903194 82.120759 \n",
       "L 163.941083 82.199462 \n",
       "L 164.980452 82.253963 \n",
       "L 166.02136 82.284233 \n",
       "L 167.063871 82.290237 \n",
       "L 168.108048 82.27194 \n",
       "L 169.153953 82.229302 \n",
       "L 170.201649 82.16228 \n",
       "L 171.251201 82.070831 \n",
       "L 172.302671 81.954906 \n",
       "L 173.356125 81.814454 \n",
       "L 174.411627 81.649422 \n",
       "L 175.469243 81.459752 \n",
       "L 176.529037 81.245385 \n",
       "L 177.591075 81.006257 \n",
       "L 178.655425 80.742303 \n",
       "L 179.722153 80.453454 \n",
       "L 180.791326 80.139638 \n",
       "L 181.863013 79.800779 \n",
       "L 182.93728 79.4368 \n",
       "L 184.014199 79.047618 \n",
       "L 185.093837 78.633148 \n",
       "L 186.176264 78.193304 \n",
       "L 187.261552 77.727993 \n",
       "L 188.349771 77.237121 \n",
       "L 189.440992 76.720591 \n",
       "L 190.535289 76.1783 \n",
       "L 191.632734 75.610144 \n",
       "L 192.733401 75.016016 \n",
       "L 193.837363 74.395803 \n",
       "L 194.944696 73.74939 \n",
       "L 196.055475 73.076659 \n",
       "L 197.169777 72.377488 \n",
       "L 198.287679 71.651751 \n",
       "L 199.409259 70.899317 \n",
       "L 200.534594 70.120055 \n",
       "L 201.663765 69.313826 \n",
       "L 202.796852 68.48049 \n",
       "L 203.933936 67.619903 \n",
       "L 205.075098 66.731915 \n",
       "L 206.220422 65.816373 \n",
       "L 207.369991 64.873122 \n",
       "L 208.52389 63.902 \n",
       "L 209.682205 62.902842 \n",
       "L 210.845021 61.87548 \n",
       "L 212.012427 60.819739 \n",
       "L 213.184511 59.735443 \n",
       "L 214.361362 58.622408 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pdebd63276f)\" d=\"M 114.713531 51.671053 \n",
       "L 115.739143 53.036316 \n",
       "L 116.763228 54.375554 \n",
       "L 117.785847 55.688888 \n",
       "L 118.80706 56.976432 \n",
       "L 119.826927 58.2383 \n",
       "L 120.845509 59.474599 \n",
       "L 121.862864 60.685437 \n",
       "L 122.879053 61.870916 \n",
       "L 123.894134 63.031136 \n",
       "L 124.908166 64.166195 \n",
       "L 125.921208 65.276186 \n",
       "L 126.933318 66.361201 \n",
       "L 127.944555 67.421326 \n",
       "L 128.954976 68.456649 \n",
       "L 129.96464 69.467249 \n",
       "L 130.973605 70.453208 \n",
       "L 131.981927 71.414601 \n",
       "L 132.989665 72.351502 \n",
       "L 133.996876 73.263981 \n",
       "L 135.003618 74.152106 \n",
       "L 136.009946 75.015943 \n",
       "L 137.015919 75.855553 \n",
       "L 138.021594 76.670997 \n",
       "L 139.027026 77.46233 \n",
       "L 140.032274 78.229607 \n",
       "L 141.037394 78.97288 \n",
       "L 142.042443 79.692195 \n",
       "L 143.047477 80.3876 \n",
       "L 144.052553 81.059137 \n",
       "L 145.057729 81.706847 \n",
       "L 146.06306 82.330768 \n",
       "L 147.068604 82.930933 \n",
       "L 148.074416 83.507377 \n",
       "L 149.080555 84.060127 \n",
       "L 150.087077 84.589212 \n",
       "L 151.094039 85.094655 \n",
       "L 152.101497 85.576478 \n",
       "L 153.10951 86.0347 \n",
       "L 154.118133 86.469337 \n",
       "L 155.127425 86.880404 \n",
       "L 156.137442 87.267911 \n",
       "L 157.148243 87.631866 \n",
       "L 158.159885 87.972274 \n",
       "L 159.172425 88.28914 \n",
       "L 160.185922 88.582464 \n",
       "L 161.200433 88.852242 \n",
       "L 162.216018 89.098471 \n",
       "L 163.232734 89.321143 \n",
       "L 164.25064 89.520246 \n",
       "L 165.269795 89.695769 \n",
       "L 166.290258 89.847695 \n",
       "L 167.312089 89.976007 \n",
       "L 168.335346 90.080682 \n",
       "L 169.360091 90.161698 \n",
       "L 170.386383 90.219027 \n",
       "L 171.414282 90.252641 \n",
       "L 172.44385 90.262507 \n",
       "L 173.475146 90.248591 \n",
       "L 174.508234 90.210855 \n",
       "L 175.543173 90.149259 \n",
       "L 176.580027 90.063759 \n",
       "L 177.618858 89.95431 \n",
       "L 178.659727 89.820862 \n",
       "L 179.7027 89.663365 \n",
       "L 180.747838 89.481763 \n",
       "L 181.795207 89.275999 \n",
       "L 182.84487 89.046013 \n",
       "L 183.896891 88.791741 \n",
       "L 184.951338 88.513117 \n",
       "L 186.008274 88.210072 \n",
       "L 187.067767 87.882534 \n",
       "L 188.129883 87.530427 \n",
       "L 189.194689 87.153673 \n",
       "L 190.262253 86.752191 \n",
       "L 191.332643 86.325896 \n",
       "L 192.405928 85.8747 \n",
       "L 193.482178 85.398513 \n",
       "L 194.561462 84.897241 \n",
       "L 195.643851 84.370785 \n",
       "L 196.729416 83.819047 \n",
       "L 197.818229 83.241921 \n",
       "L 198.910363 82.639302 \n",
       "L 200.00589 82.011077 \n",
       "L 201.104884 81.357134 \n",
       "L 202.207421 80.677354 \n",
       "L 203.313575 79.971617 \n",
       "L 204.423422 79.239799 \n",
       "L 205.537039 78.481771 \n",
       "L 206.654504 77.697402 \n",
       "L 207.775894 76.886556 \n",
       "L 208.90129 76.049095 \n",
       "L 210.030771 75.184875 \n",
       "L 211.164417 74.293751 \n",
       "L 212.302311 73.37557 \n",
       "L 213.444536 72.43018 \n",
       "L 214.591174 71.457422 \n",
       "L 215.742311 70.457133 \n",
       "L 216.898031 69.429147 \n",
       "L 218.058421 68.373293 \n",
       "L 219.22357 67.289397 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pdebd63276f\">\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"7.2\" y=\"7.2\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x, y = np.mgrid[-1: 1: 101j, -1: 1: 101j]\n",
    "z = x**2 - y**2\n",
    "ax = d2l.plt.figure().add_subplot(111, projection='3d')\n",
    "ax.plot_wireframe(x, y, z, **{'rstride': 10, 'cstride': 10})\n",
    "ax.plot([0], [0], [0], 'rx')\n",
    "ticks = [-1, 0, 1]\n",
    "d2l.plt.xticks(ticks)\n",
    "d2l.plt.yticks(ticks)\n",
    "ax.set_zticks(ticks)\n",
    "d2l.plt.xlabel('x')\n",
    "d2l.plt.ylabel('y');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume that the input of a function is a $k$-dimensional vector and its\n",
    "output is a scalar, so its Hessian matrix will have $k$ eigenvalues\n",
    "(refer to Section 16.2).\n",
    "The solution of the\n",
    "function could be a local minimum, a local maximum, or a saddle point at a\n",
    "position where the function gradient is zero:\n",
    "\n",
    "* When the eigenvalues of the function's Hessian matrix at the zero-gradient position are all positive, we have a local minimum for the function.\n",
    "* When the eigenvalues of the function's Hessian matrix at the zero-gradient position are all negative, we have a local maximum for the function.\n",
    "* When the eigenvalues of the function's Hessian matrix at the zero-gradient position are negative and positive, we have a saddle point for the function.\n",
    "\n",
    "For high-dimensional problems the likelihood that at least some of the eigenvalues are negative is quite high. This makes saddle points more likely than local minima. We will discuss some exceptions to this situation in the next section when introducing convexity. In short, convex functions are those where the eigenvalues of the Hessian are never negative. Sadly, though, most deep learning problems do not fall into this category. Nonetheless it's a great tool to study optimization algorithms.\n",
    "\n",
    "### Vanishing Gradients\n",
    "\n",
    "Probably the most insidious problem to encounter are vanishing gradients. For instance, assume that we want to minimize the function $f(x) = \\tanh(x)$ and we happen to get started at $x = 4$. As we can see, the gradient of $f$ is close to nil. More specifically $f'(x) = 1 - \\tanh^2(x)$ and thus $f'(4) = 0.0013$. Consequently optimization will get stuck for a long time before we make progress. This turns out to be one of the reasons that training deep learning models was quite tricky prior to the introduction of the ReLu activation function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 313.960938 184.15625\" width=\"313.960938pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M -0 184.15625 \n",
       "L 313.960938 184.15625 \n",
       "L 313.960938 -0 \n",
       "L -0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 52.160938 146.6 \n",
       "L 303.260938 146.6 \n",
       "L 303.260938 10.7 \n",
       "L 52.160938 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m0bc0d434ca\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"63.574574\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-8722\"/>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(56.20348 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"96.231616\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −1 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(88.860523 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"128.888659\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(125.707409 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"161.545701\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(158.364451 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"194.202744\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(191.021494 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"226.859787\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 3 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-51\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(223.678537 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"259.516829\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-52\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(256.335579 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"292.173872\" xlink:href=\"#m0bc0d434ca\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-53\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(288.992622 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-120\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(174.751563 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-120\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m60c2848016\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m60c2848016\" y=\"142.685648\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- −1.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-46\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 146.484867)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m60c2848016\" y=\"111.232097\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- −0.5 -->\n",
       "      <g transform=\"translate(20.878125 115.031316)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-8722\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m60c2848016\" y=\"79.778547\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(29.257812 83.577766)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m60c2848016\" y=\"48.324996\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0.5 -->\n",
       "      <g transform=\"translate(29.257812 52.124215)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m60c2848016\" y=\"16.871446\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(29.257812 20.670664)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-102\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"DejaVuSans-40\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "z\n",
       "\" id=\"DejaVuSans-41\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798438 87.271094)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-102\"/>\n",
       "      <use x=\"35.205078\" xlink:href=\"#DejaVuSans-40\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#DejaVuSans-120\"/>\n",
       "      <use x=\"133.398438\" xlink:href=\"#DejaVuSans-41\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_14\">\n",
       "    <path clip-path=\"url(#p96f2acb563)\" d=\"M 63.574574 140.422727 \n",
       "L 68.47313 139.650126 \n",
       "L 73.045116 138.700329 \n",
       "L 76.963961 137.662474 \n",
       "L 80.556236 136.487285 \n",
       "L 83.82194 135.196637 \n",
       "L 87.087644 133.65753 \n",
       "L 90.026778 132.027856 \n",
       "L 92.965912 130.13562 \n",
       "L 95.905046 127.950416 \n",
       "L 98.517609 125.738091 \n",
       "L 101.130173 123.251724 \n",
       "L 103.742736 120.475003 \n",
       "L 106.3553 117.395861 \n",
       "L 109.294433 113.562778 \n",
       "L 112.233567 109.341437 \n",
       "L 115.499271 104.216237 \n",
       "L 118.764976 98.678174 \n",
       "L 122.683821 91.589116 \n",
       "L 128.235518 81.036521 \n",
       "L 136.073208 66.158023 \n",
       "L 139.992053 59.177893 \n",
       "L 143.584328 53.238103 \n",
       "L 146.850032 48.292271 \n",
       "L 149.789166 44.242353 \n",
       "L 152.7283 40.583312 \n",
       "L 155.667434 37.311858 \n",
       "L 158.279997 34.718324 \n",
       "L 160.892561 32.405354 \n",
       "L 163.831694 30.11583 \n",
       "L 166.770828 28.129308 \n",
       "L 169.709962 26.415482 \n",
       "L 172.649096 24.944173 \n",
       "L 175.9148 23.558613 \n",
       "L 179.507075 22.294951 \n",
       "L 183.09935 21.26128 \n",
       "L 187.344765 20.283491 \n",
       "L 191.916751 19.467407 \n",
       "L 197.141878 18.767213 \n",
       "L 203.346716 18.174093 \n",
       "L 210.857836 17.696937 \n",
       "L 220.328378 17.334975 \n",
       "L 233.391195 17.080146 \n",
       "L 253.965132 16.930715 \n",
       "L 291.847301 16.877273 \n",
       "L 291.847301 16.877273 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 52.160938 146.6 \n",
       "L 52.160938 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 303.260938 146.6 \n",
       "L 303.260938 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 52.160938 146.6 \n",
       "L 303.260938 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 52.160938 10.7 \n",
       "L 303.260938 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 243.643103 68.26995 \n",
       "Q 251.28495 43.525949 258.596882 19.850198 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "    <path d=\"M 255.505602 23.081915 \n",
       "L 258.596882 19.850198 \n",
       "L 259.327487 24.262252 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:round;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_16\">\n",
       "    <!-- vanishing gradient -->\n",
       "    <defs>\n",
       "     <path d=\"M 2.984375 54.6875 \n",
       "L 12.5 54.6875 \n",
       "L 29.59375 8.796875 \n",
       "L 46.6875 54.6875 \n",
       "L 56.203125 54.6875 \n",
       "L 35.6875 0 \n",
       "L 23.484375 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-118\"/>\n",
       "     <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "     <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-110\"/>\n",
       "     <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-105\"/>\n",
       "     <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-115\"/>\n",
       "     <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-104\"/>\n",
       "     <path d=\"M 45.40625 27.984375 \n",
       "Q 45.40625 37.75 41.375 43.109375 \n",
       "Q 37.359375 48.484375 30.078125 48.484375 \n",
       "Q 22.859375 48.484375 18.828125 43.109375 \n",
       "Q 14.796875 37.75 14.796875 27.984375 \n",
       "Q 14.796875 18.265625 18.828125 12.890625 \n",
       "Q 22.859375 7.515625 30.078125 7.515625 \n",
       "Q 37.359375 7.515625 41.375 12.890625 \n",
       "Q 45.40625 18.265625 45.40625 27.984375 \n",
       "z\n",
       "M 54.390625 6.78125 \n",
       "Q 54.390625 -7.171875 48.1875 -13.984375 \n",
       "Q 42 -20.796875 29.203125 -20.796875 \n",
       "Q 24.46875 -20.796875 20.265625 -20.09375 \n",
       "Q 16.0625 -19.390625 12.109375 -17.921875 \n",
       "L 12.109375 -9.1875 \n",
       "Q 16.0625 -11.328125 19.921875 -12.34375 \n",
       "Q 23.78125 -13.375 27.78125 -13.375 \n",
       "Q 36.625 -13.375 41.015625 -8.765625 \n",
       "Q 45.40625 -4.15625 45.40625 5.171875 \n",
       "L 45.40625 9.625 \n",
       "Q 42.625 4.78125 38.28125 2.390625 \n",
       "Q 33.9375 0 27.875 0 \n",
       "Q 17.828125 0 11.671875 7.65625 \n",
       "Q 5.515625 15.328125 5.515625 27.984375 \n",
       "Q 5.515625 40.671875 11.671875 48.328125 \n",
       "Q 17.828125 56 27.875 56 \n",
       "Q 33.9375 56 38.28125 53.609375 \n",
       "Q 42.625 51.21875 45.40625 46.390625 \n",
       "L 45.40625 54.6875 \n",
       "L 54.390625 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-103\"/>\n",
       "     <path id=\"DejaVuSans-32\"/>\n",
       "     <path d=\"M 41.109375 46.296875 \n",
       "Q 39.59375 47.171875 37.8125 47.578125 \n",
       "Q 36.03125 48 33.890625 48 \n",
       "Q 26.265625 48 22.1875 43.046875 \n",
       "Q 18.109375 38.09375 18.109375 28.8125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 20.953125 51.171875 25.484375 53.578125 \n",
       "Q 30.03125 56 36.53125 56 \n",
       "Q 37.453125 56 38.578125 55.875 \n",
       "Q 39.703125 55.765625 41.0625 55.515625 \n",
       "z\n",
       "\" id=\"DejaVuSans-114\"/>\n",
       "     <path d=\"M 45.40625 46.390625 \n",
       "L 45.40625 75.984375 \n",
       "L 54.390625 75.984375 \n",
       "L 54.390625 0 \n",
       "L 45.40625 0 \n",
       "L 45.40625 8.203125 \n",
       "Q 42.578125 3.328125 38.25 0.953125 \n",
       "Q 33.9375 -1.421875 27.875 -1.421875 \n",
       "Q 17.96875 -1.421875 11.734375 6.484375 \n",
       "Q 5.515625 14.40625 5.515625 27.296875 \n",
       "Q 5.515625 40.1875 11.734375 48.09375 \n",
       "Q 17.96875 56 27.875 56 \n",
       "Q 33.9375 56 38.25 53.625 \n",
       "Q 42.578125 51.265625 45.40625 46.390625 \n",
       "z\n",
       "M 14.796875 27.296875 \n",
       "Q 14.796875 17.390625 18.875 11.75 \n",
       "Q 22.953125 6.109375 30.078125 6.109375 \n",
       "Q 37.203125 6.109375 41.296875 11.75 \n",
       "Q 45.40625 17.390625 45.40625 27.296875 \n",
       "Q 45.40625 37.203125 41.296875 42.84375 \n",
       "Q 37.203125 48.484375 30.078125 48.484375 \n",
       "Q 22.953125 48.484375 18.875 42.84375 \n",
       "Q 14.796875 37.203125 14.796875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-100\"/>\n",
       "     <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-101\"/>\n",
       "     <path d=\"M 18.3125 70.21875 \n",
       "L 18.3125 54.6875 \n",
       "L 36.8125 54.6875 \n",
       "L 36.8125 47.703125 \n",
       "L 18.3125 47.703125 \n",
       "L 18.3125 18.015625 \n",
       "Q 18.3125 11.328125 20.140625 9.421875 \n",
       "Q 21.96875 7.515625 27.59375 7.515625 \n",
       "L 36.8125 7.515625 \n",
       "L 36.8125 0 \n",
       "L 27.59375 0 \n",
       "Q 17.1875 0 13.234375 3.875 \n",
       "Q 9.28125 7.765625 9.28125 18.015625 \n",
       "L 9.28125 47.703125 \n",
       "L 2.6875 47.703125 \n",
       "L 2.6875 54.6875 \n",
       "L 9.28125 54.6875 \n",
       "L 9.28125 70.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-116\"/>\n",
       "    </defs>\n",
       "    <g transform=\"translate(194.202744 79.778547)scale(0.1 -0.1)\">\n",
       "     <use xlink:href=\"#DejaVuSans-118\"/>\n",
       "     <use x=\"59.179688\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"120.458984\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"183.837891\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"211.621094\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     <use x=\"263.720703\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     <use x=\"327.099609\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"354.882812\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"418.261719\" xlink:href=\"#DejaVuSans-103\"/>\n",
       "     <use x=\"481.738281\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"513.525391\" xlink:href=\"#DejaVuSans-103\"/>\n",
       "     <use x=\"577.001953\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"618.115234\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"679.394531\" xlink:href=\"#DejaVuSans-100\"/>\n",
       "     <use x=\"742.871094\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"770.654297\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"832.177734\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"895.556641\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p96f2acb563\">\n",
       "   <rect height=\"135.9\" width=\"251.1\" x=\"52.160938\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 324x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-2.0, 5.0, 0.01)\n",
    "fig, = d2l.plt.plot(x, np.tanh(x))\n",
    "fig.axes.annotate('vanishing gradient', xy=(4, 1), xytext=(2, 0.0),\n",
    "arrowprops=dict(arrowstyle='->'))\n",
    "d2l.plt.xlabel('x')\n",
    "d2l.plt.ylabel('f(x)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw, optimization for deep learning is full of challenges. Fortunately there exists a robust range of algorithms that perform well and that are easy to use even for beginners. Furthermore, it isn't really necessary to find *the* best solution. Local optima or even approximate solutions thereof are still very useful.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Minimizing the training error does *not* guarantee that we find the best set of parameters to minimize the expected error.\n",
    "* The optimization problems may have many local minima.\n",
    "* The problem may have even more saddle points, as generally the problems are not convex.\n",
    "* Vanishing gradients can cause optimization to stall. Often a reparametrization of the problem helps. Good initialization of the parameters can be beneficial, too.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Consider a simple multilayer perceptron with a single hidden layer of, say, $d$ dimensions in the hidden layer and a single output. Show that for any local minimum there are at least $d!$ equivalent solutions that behave identically.\n",
    "1. Assume that we have a symmetric random matrix $\\mathbf{M}$ where the entries $M_{ij} = M_{ji}$ are each drawn from some probability distribution $p_{ij}$. Furthermore assume that $p_{ij}(x) = p_{ij}(-x)$, i.e. that the distribution is symmetric (see e.g. [1] for details).\n",
    "    * Prove that the distribution over eigenvalues is also symmetric. That is, for any eigenvector $\\mathbf{v}$ the probability that the associated eigenvalue $\\lambda$ satisfies $\\Pr(\\lambda > 0) = \\Pr(\\lambda < 0)$.\n",
    "    * Why does the above *not* imply $\\Pr(\\lambda > 0) = 0.5$?\n",
    "1. What other challenges involved in deep learning optimization can you think of?\n",
    "1. Assume that you want to balance a (real) ball on a (real) saddle.\n",
    "    * Why is this hard?\n",
    "    * Can you exploit this effect also for optimization algorithms?\n",
    "    \n",
    "## References\n",
    "\n",
    "[1] Wigner, E. P. (1958). On the distribution of the roots of certain symmetric matrices. Annals of Mathematics, 325-327."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}