notebooks/en/vmOptimization-or-tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to optimize target VM costs when moving them to Azure using an SMT solver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scenario\n",
    "\n",
    "[Last time](/vmOptimization-basics.ipynb) we optimized the total VM cost in the Azure Cloud using minizinc. This time we want to avoid installing additional software and use a solver directly, from .NET. For this experiment, we want to use [or-tools](https://developers.google.com/optimization/introduction/overview) - the SMT solver from Google.\n",
    "\n",
    "\n",
    "> OR-Tools is open source software for combinatorial optimization, which seeks to find the best solution to a problem out of a very large set of possible solutions. Here are some examples of problems that OR-Tools solves:\n",
    ">\n",
    "> - Vehicle routing: Find optimal routes for vehicle fleets that pick up and deliver packages given constraints (e.g., \"this truck can't hold more than 20,000 pounds\" or \"all deliveries must be made within a two-hour window\").\n",
    "> - Scheduling: Find the optimal schedule for a complex set of tasks, some of which need to be performed before others, on a fixed set of machines, or other resources.\n",
    "> - Bin packing: Pack as many objects of various sizes as possible into a fixed number of bins with maximum capacities.\n",
    "\n",
    "One of the things here is that or-tools has a .NET interface, so we can use it in Powershell without having to install anything in addition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> There is a small issue, though. The solver is C, I guess. The .NET interface is just a wrapper. For it to work it needs a [vc runtime](https://docs.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170). But the Powershell module we are going to use already includes required libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify stuff, I made a simple PowerShell module [AzureVmCalc](https://github.com/eosfor/AzureVmCalc). It contains everything needed to start working."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> At the moment, the module is still alpha, but it is working and can be used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installing it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    }
   },
   "outputs": [],
   "source": [
    "Install-Module AzureVmCalc -AllowPrerelease -Scope CurrentUser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    }
   },
   "outputs": [],
   "source": [
    "Import-Module AzureVmCalc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    }
   },
   "outputs": [],
   "source": [
    "$sourceVMs = Import-Csv \".\\vm-optimization-ortools\\vmdata.csv\"\n",
    "$sourceVMs  | % { $_.cpu = [int]$_.cpu; $_.ram = [int]$_.ram; $_.datadisk = [int]$_.datadisk; }\n",
    "\n",
    "$targetSizes = import-csv \".\\vm-optimization-ortools\\vmCostACUData.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `vmdata.csv` file contains information about the \"source\" server sizes. The `vmCostACUData` contains data about VM sizes and their price from an Azure region"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start the commandlet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    }
   },
   "outputs": [],
   "source": [
    "$x = Start-OrToolsModelCalculation -SourceVM $sourceVMs -TargetVM $targetSizes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The commandlet takes the list of source servers, the list of target sizes with their attributes and returns an array. Under the hood, processing goes in two stages. First, the model tries to maximize the total performance of the whole set of VMs, by using the ACU metric, supplied by Microsoft. Then it pins the performance objective and tries to minimize the total cost, keeping the performance objective intact. In the end, the output contains the results of both calculations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\u001b[32;1mVmMappingResult\u001b[0m\r\n",
      "\u001b[32;1m---------------\u001b[0m\r\n",
      "{Standard_E96-48as_v4, Standard_E96-48as_v4, Standard_E96-48as_v4, Standard_E96-48as_v4…}\r\n",
      "{Standard_E4-2as_v4, Standard_E4-2as_v4, Standard_E4-2as_v4, Standard_E8-4as_v4…}\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "$x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table> <colgroup><col/><col/><col/><col/><col/><col/><col/><col/><col/></colgroup> <tr><th>vmid</th><th>cpu</th><th>ram</th><th>Name</th><th>vCPUs</th><th>MemoryGB</th><th>vCPUsPerCore</th><th>ACUs</th><th>retailPrice</th></tr> <tr><td>vmN1</td><td>2</td><td>12</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN2</td><td>2</td><td>24</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN3</td><td>2</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN4</td><td>6</td><td>23</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN5</td><td>3</td><td>16</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN6</td><td>2</td><td>32</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN7</td><td>2</td><td>7</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN8</td><td>3</td><td>32</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN9</td><td>2</td><td>8</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN10</td><td>1</td><td>40</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN11</td><td>2</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN12</td><td>3</td><td>48</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN13</td><td>4</td><td>8</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN14</td><td>2</td><td>16</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN15</td><td>10</td><td>27</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN16</td><td>4</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN17</td><td>1</td><td>20</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN18</td><td>3</td><td>25</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN19</td><td>5</td><td>30</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN20</td><td>5</td><td>35</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN21</td><td>4</td><td>48</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN22</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN23</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN24</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN25</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN26</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN27</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN28</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN29</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN30</td><td>2</td><td>27</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN31</td><td>8</td><td>90</td><td>Standard_E16-8as_v4</td><td>16</td><td>128</td><td>2</td><td>230</td><td>1.744</td></tr> <tr><td>vmN32</td><td>7</td><td>180</td><td>Standard_E32-8as_v4</td><td>32</td><td>256</td><td>2</td><td>230</td><td>3.488</td></tr> <tr><td>vmN33</td><td>6</td><td>4</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN34</td><td>4</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN35</td><td>5</td><td>48</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN36</td><td>2</td><td>26</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN37</td><td>2</td><td>26</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN38</td><td>2</td><td>26</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN39</td><td>2</td><td>26</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN40</td><td>5</td><td>50</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN41</td><td>2</td><td>32</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN42</td><td>6</td><td>60</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN43</td><td>6</td><td>58</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN44</td><td>7</td><td>98</td><td>Standard_E16-8as_v4</td><td>16</td><td>128</td><td>2</td><td>230</td><td>1.744</td></tr> <tr><td>vmN45</td><td>2</td><td>15</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN46</td><td>2</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN47</td><td>2</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN48</td><td>2</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN49</td><td>8</td><td>20</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN50</td><td>6</td><td>32</td><td>Standard_E8-2as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN51</td><td>2</td><td>32</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN52</td><td>5</td><td>20</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN53</td><td>6</td><td>50</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN54</td><td>3</td><td>40</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN55</td><td>8</td><td>46</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN56</td><td>1</td><td>25</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN57</td><td>3</td><td>16</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN58</td><td>2</td><td>12</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN59</td><td>2</td><td>12</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN60</td><td>8</td><td>4</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN61</td><td>4</td><td>35</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN62</td><td>8</td><td>72</td><td>Standard_E16-8as_v4</td><td>16</td><td>128</td><td>2</td><td>230</td><td>1.744</td></tr> <tr><td>vmN63</td><td>2</td><td>15</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN64</td><td>2</td><td>15</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN65</td><td>2</td><td>15</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN66</td><td>2</td><td>15</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN67</td><td>6</td><td>160</td><td>Standard_E32-8as_v4</td><td>32</td><td>256</td><td>2</td><td>230</td><td>3.488</td></tr> <tr><td>vmN68</td><td>2</td><td>4</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN69</td><td>2</td><td>20</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN70</td><td>2</td><td>31</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN71</td><td>2</td><td>31</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN72</td><td>4</td><td>24</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN73</td><td>2</td><td>32</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN74</td><td>6</td><td>50</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN75</td><td>3</td><td>65</td><td>Standard_E16-8as_v4</td><td>16</td><td>128</td><td>2</td><td>230</td><td>1.744</td></tr> <tr><td>vmN76</td><td>2</td><td>16</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN77</td><td>2</td><td>6</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN78</td><td>8</td><td>36</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN79</td><td>3</td><td>50</td><td>Standard_E8-4as_v4</td><td>8</td><td>64</td><td>2</td><td>230</td><td>0.872</td></tr> <tr><td>vmN80</td><td>2</td><td>11</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN81</td><td>4</td><td>24</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN82</td><td>4</td><td>30</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN83</td><td>2</td><td>16</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN84</td><td>1</td><td>18</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN85</td><td>3</td><td>24</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN86</td><td>2</td><td>20</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> <tr><td>vmN87</td><td>2</td><td>32</td><td>Standard_E4-2as_v4</td><td>4</td><td>32</td><td>2</td><td>230</td><td>0.436</td></tr> </table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "$view = $x[1].VmMappingResult | select vmid, cpu, ram, name, vCPUs, MemoryGB, vCPUsPerCore, ACUs, retailPrice | ConvertTo-Html -Fragment\n",
    "[Microsoft.DotNet.Interactive.Kernel]::HTML($view) | Out-Display"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "C#"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}