ml_trials.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# i. BUSINESS JOB DESCRIPTION\n",
    "\n",
    "*   Data Money is a company that offers data analysis consultancy services, guided towards other partner companies worldwide.\n",
    "*   The main feature on Data Money services resides on the metodology used on training and tunning machine learning models, a task properly achieved by their team of Data Scientists, clarifying the behaviour of each algorithm with proven explained results.\n",
    "\n",
    "# ii. THE CHALLENGE\n",
    "\n",
    "*   In order to keep the company's data scientists team growth in performance, Data Money required the execution of a higher number of model trials, with goals of further understanding which were the adequate scenarios to apply each algorithm. These choices are then expected to be based upon the showcasing of verified metrics, serving as means of comparisons between machine learning models to know when their performance is best optimized/minimized.\n",
    "*   Being a Data Scientist recently hired by the company, I was then tasked of realizing trials for Classification, Regression and clustering machine learning models, to further report their results to the other working teams.\n",
    "\n",
    "# iii. BUSINESS CHALLANGE SPECIFICATIONS\n",
    "\n",
    "*   It is expected to verify the performance results on 3 different datasets, those being:\n",
    "    1. The training dataset (Used for both training and model prediction)\n",
    "    2. Validation Dataset\n",
    "    3. Test Dataset\n",
    "\n",
    "\n",
    "*   The results will then be reported as a table for each different dataset, containing the comparison of metrics for a selection of machine learning models.\n",
    "*   The only exception will be the analysis for clustering model: Given that it is an unsupervized learning model, it will then be only one table containing its results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0.0 IMPORTS AND HELPER FUNCTIONS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import warnings\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "from sklearn.neighbors      import KNeighborsClassifier, KNeighborsRegressor\n",
    "from sklearn.tree           import DecisionTreeClassifier, DecisionTreeRegressor\n",
    "from sklearn.ensemble       import RandomForestClassifier, RandomForestRegressor\n",
    "from sklearn.linear_model   import LogisticRegression, LinearRegression, Lasso, Ridge, ElasticNet\n",
    "\n",
    "from sklearn                import preprocessing as pp\n",
    "from sklearn                import cluster as c\n",
    "from sklearn                import metrics as mt\n",
    "from sklearn                import model_selection as ms \n",
    "from sklearn                import datasets as dt\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.0 CLASSIFICATION MODELS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   The main metric used for evaluation on overall model performance will be the f1_score, since it considers an equilibrium on both precision and recall metrics.\n",
    "*   Accuracy, precision and recall scores will also be listed, as well as the hyperparameter set tunned for each iteration, in order to obtain insights on how each model works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_training_classification = pd.read_csv('datasets\\classification\\X_training.csv')\n",
    "y_training_classification = pd.read_csv('datasets\\classification\\y_training.csv')\n",
    "\n",
    "X_validation_classification = pd.read_csv('datasets\\classification\\X_validation.csv')\n",
    "y_validation_classification = pd.read_csv('datasets\\classification\\y_validation.csv')\n",
    "\n",
    "X_test_classification = pd.read_csv('datasets\\classification\\X_test.csv')\n",
    "y_test_classification = pd.read_csv('datasets\\classification\\y_test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training dataset: rows = 72515, columns = 25\n",
      "Validation dataset: rows = 31079, columns = 25\n",
      "Test dataset: rows = 25893, columns = 25\n",
      "Number of classes:2, Distinct Classes:[0 1]\n"
     ]
    }
   ],
   "source": [
    "print(f'Training dataset: rows = {X_training_classification.shape[0]}, columns = {X_training_classification.shape[1]}')\n",
    "print(f'Validation dataset: rows = {X_validation_classification.shape[0]}, columns = {X_validation_classification.shape[1]}')\n",
    "print(f'Test dataset: rows = {X_test_classification.shape[0]}, columns = {X_test_classification.shape[1]}')\n",
    "\n",
    "print(f'Number of classes:{len(np.unique(y_training_classification))}, Distinct Classes:{np.unique(y_training_classification)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Every class variable array ('y' variables), matches in size with their features counterparts('x' variables).\n",
    "*   The datasets are presented as being binary classed, with '0' and '1' as the class variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "neighbours_list = list()\n",
    "\n",
    "acc_list_training = list()\n",
    "precision_list_training = list()\n",
    "recall_list_training = list()\n",
    "f1_list_training = list()\n",
    "\n",
    "acc_list_validation = list()\n",
    "precision_list_validation = list()\n",
    "recall_list_validation = list()\n",
    "f1_list_validation = list()\n",
    "\n",
    "acc_list_test = list()\n",
    "precision_list_test = list()\n",
    "recall_list_test = list()\n",
    "f1_list_test = list()\n",
    "\n",
    "for i in range(3, 15, 2):\n",
    "    neighbours_list.append(i)\n",
    "    #Model Definition\n",
    "    knn_classifier = KNeighborsClassifier(n_neighbors = i)\n",
    "    \n",
    "    #Model Train\n",
    "    knn_classifier.fit(X_training_classification, np.ravel(y_training_classification))\n",
    "    \n",
    "    #Model Predict\n",
    "    yhat_knn_training   = knn_classifier.predict(X_training_classification)\n",
    "    yhat_knn_validation = knn_classifier.predict(X_validation_classification)\n",
    "    yhat_knn_test       = knn_classifier.predict(X_test_classification)\n",
    "    \n",
    "    #training scores\n",
    "    acc_score_training       = mt.accuracy_score(y_training_classification, yhat_knn_training)\n",
    "    precision_score_training = mt.precision_score(y_training_classification, yhat_knn_training)\n",
    "    recall_score_training    = mt.recall_score(y_training_classification, yhat_knn_training)\n",
    "    f1_score_training        = mt.f1_score(y_training_classification, yhat_knn_training)\n",
    "    \n",
    "    acc_list_training.append(acc_score_training)\n",
    "    precision_list_training.append(precision_score_training)\n",
    "    recall_list_training.append(recall_score_training)\n",
    "    f1_list_training.append(f1_score_training)\n",
    "    \n",
    "    #validation scores\n",
    "    acc_score_validation       = mt.accuracy_score(y_validation_classification, yhat_knn_validation)\n",
    "    precision_score_validation = mt.precision_score(y_validation_classification, yhat_knn_validation)\n",
    "    recall_score_validation    = mt.recall_score(y_validation_classification, yhat_knn_validation)\n",
    "    f1_score_validation        = mt.f1_score(y_validation_classification, yhat_knn_validation)\n",
    "    \n",
    "    acc_list_validation.append(acc_score_validation)\n",
    "    precision_list_validation.append(precision_score_validation)\n",
    "    recall_list_validation.append(recall_score_validation)\n",
    "    f1_list_validation.append(f1_score_validation)\n",
    "    \n",
    "    #test scores\n",
    "    acc_score_test       = mt.accuracy_score(y_test_classification, yhat_knn_test)\n",
    "    precision_score_test = mt.precision_score(y_test_classification, yhat_knn_test)\n",
    "    recall_score_test    = mt.recall_score(y_test_classification, yhat_knn_test)\n",
    "    f1_score_test        = mt.f1_score(y_test_classification, yhat_knn_test)\n",
    "    \n",
    "    acc_list_test.append(acc_score_test)\n",
    "    precision_list_test.append(precision_score_test)\n",
    "    recall_list_test.append(recall_score_test)\n",
    "    f1_list_test.append(f1_score_test)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neighbors</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>0.832186</td>\n",
       "      <td>0.812008</td>\n",
       "      <td>0.79741</td>\n",
       "      <td>0.804643</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   neighbors  accuracy_score  precision_score  recall_score  f1_score\n",
       "0          3        0.832186         0.812008       0.79741  0.804643"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_training_df = pd.DataFrame({'neighbors': neighbours_list, 'accuracy_score':acc_list_training, \n",
    "                                'precision_score': precision_list_training, 'recall_score':recall_list_training, \n",
    "                                'f1_score':f1_list_training})\n",
    "\n",
    "knn_training_max = knn_training_df.query('f1_score == f1_score.max()')\n",
    "knn_training_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neighbors</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>0.676277</td>\n",
       "      <td>0.627851</td>\n",
       "      <td>0.621278</td>\n",
       "      <td>0.624548</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   neighbors  accuracy_score  precision_score  recall_score  f1_score\n",
       "0          3        0.676277         0.627851      0.621278  0.624548"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_validation_df = pd.DataFrame({'neighbors': neighbours_list, 'accuracy_score':acc_list_validation, \n",
    "                                'precision_score': precision_list_validation, 'recall_score':recall_list_validation, \n",
    "                                'f1_score':f1_list_validation})\n",
    "\n",
    "knn_validation_max = knn_validation_df.query('f1_score == f1_score.max()')\n",
    "knn_validation_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neighbors</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>0.672228</td>\n",
       "      <td>0.630462</td>\n",
       "      <td>0.611879</td>\n",
       "      <td>0.621031</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   neighbors  accuracy_score  precision_score  recall_score  f1_score\n",
       "0          3        0.672228         0.630462      0.611879  0.621031"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_test_df = pd.DataFrame({'neighbors': neighbours_list, 'accuracy_score':acc_list_test, \n",
    "                            'precision_score': precision_list_test, 'recall_score':recall_list_test, \n",
    "                            'f1_score':f1_list_test})\n",
    "\n",
    "knn_test_max = knn_test_df.query('f1_score == f1_score.max()')\n",
    "knn_test_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   All datasets presented optimal results for lower number of neighbors, justifying the choice for n_neighbors=3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_list = list()\n",
    "\n",
    "acc_list_training = list()\n",
    "precision_list_training = list()\n",
    "recall_list_training = list()\n",
    "f1_list_training = list()\n",
    "\n",
    "acc_list_validation = list()\n",
    "precision_list_validation = list()\n",
    "recall_list_validation = list()\n",
    "f1_list_validation = list()\n",
    "\n",
    "acc_list_test = list()\n",
    "precision_list_test = list()\n",
    "recall_list_test = list()\n",
    "f1_list_test = list()\n",
    "\n",
    "for i in range(1, 20, 2):\n",
    "    max_depth_list.append(i)\n",
    "    #Model Define\n",
    "    model_tree = DecisionTreeClassifier(max_depth=i, random_state=42)\n",
    "    \n",
    "    #Model Training\n",
    "    model_tree.fit(X_training_classification, np.ravel(y_training_classification))\n",
    "    \n",
    "    #Model Predict\n",
    "    yhat_tree_training   = model_tree.predict(X_training_classification)\n",
    "    yhat_tree_validation = model_tree.predict(X_validation_classification)\n",
    "    yhat_tree_test       = model_tree.predict(X_test_classification)\n",
    "    \n",
    "    #training scores\n",
    "    acc_score_training       = mt.accuracy_score(y_training_classification, yhat_tree_training)\n",
    "    precision_score_training = mt.precision_score(y_training_classification, yhat_tree_training)\n",
    "    recall_score_training    = mt.recall_score(y_training_classification, yhat_tree_training)\n",
    "    f1_score_training        = mt.f1_score(y_training_classification, yhat_tree_training)\n",
    "    \n",
    "    acc_list_training.append(acc_score_training)\n",
    "    precision_list_training.append(precision_score_training)\n",
    "    recall_list_training.append(recall_score_training)\n",
    "    f1_list_training.append(f1_score_training)\n",
    "    \n",
    "    #validation scores\n",
    "    acc_score_validation       = mt.accuracy_score(y_validation_classification, yhat_tree_validation)\n",
    "    precision_score_validation = mt.precision_score(y_validation_classification, yhat_tree_validation)\n",
    "    recall_score_validation    = mt.recall_score(y_validation_classification, yhat_tree_validation)\n",
    "    f1_score_validation        = mt.f1_score(y_validation_classification, yhat_tree_validation)\n",
    "    \n",
    "    acc_list_validation.append(acc_score_validation)\n",
    "    precision_list_validation.append(precision_score_validation)\n",
    "    recall_list_validation.append(recall_score_validation)\n",
    "    f1_list_validation.append(f1_score_validation)\n",
    "    \n",
    "    #test scores\n",
    "    acc_score_test       = mt.accuracy_score(y_test_classification, yhat_tree_test)\n",
    "    precision_score_test = mt.precision_score(y_test_classification, yhat_tree_test)\n",
    "    recall_score_test    = mt.recall_score(y_test_classification, yhat_tree_test)\n",
    "    f1_score_test        = mt.f1_score(y_test_classification, yhat_tree_test)\n",
    "    \n",
    "    acc_list_test.append(acc_score_test)\n",
    "    precision_list_test.append(precision_score_test)\n",
    "    recall_list_test.append(recall_score_test)\n",
    "    f1_list_test.append(f1_score_test)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>19</td>\n",
       "      <td>0.989712</td>\n",
       "      <td>0.993089</td>\n",
       "      <td>0.983104</td>\n",
       "      <td>0.988072</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  accuracy_score  precision_score  recall_score  f1_score\n",
       "9         19        0.989712         0.993089      0.983104  0.988072"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_training_df = pd.DataFrame({'max_depth': max_depth_list, 'accuracy_score':acc_list_training, \n",
    "                                'precision_score': precision_list_training, 'recall_score':recall_list_training, \n",
    "                                'f1_score':f1_list_training})\n",
    "\n",
    "tree_training_max = tree_training_df.query('f1_score == f1_score.max()')\n",
    "tree_training_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>13</td>\n",
       "      <td>0.952315</td>\n",
       "      <td>0.9563</td>\n",
       "      <td>0.932586</td>\n",
       "      <td>0.944294</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  accuracy_score  precision_score  recall_score  f1_score\n",
       "6         13        0.952315           0.9563      0.932586  0.944294"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_validation_df = pd.DataFrame({'max_depth': max_depth_list, 'accuracy_score':acc_list_validation, \n",
    "                                   'precision_score': precision_list_validation, 'recall_score':recall_list_validation, \n",
    "                                   'f1_score':f1_list_validation})\n",
    "\n",
    "tree_validation_max = tree_validation_df.query('f1_score == f1_score.max()')\n",
    "tree_validation_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>13</td>\n",
       "      <td>0.951802</td>\n",
       "      <td>0.953881</td>\n",
       "      <td>0.935416</td>\n",
       "      <td>0.944558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  accuracy_score  precision_score  recall_score  f1_score\n",
       "6         13        0.951802         0.953881      0.935416  0.944558"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_test_df = pd.DataFrame({'max_depth': max_depth_list, 'accuracy_score':acc_list_test, \n",
    "                                'precision_score': precision_list_test, 'recall_score':recall_list_test, \n",
    "                                'f1_score':f1_list_test})\n",
    "\n",
    "tree_test_max = tree_test_df.query('f1_score == f1_score.max()')\n",
    "tree_test_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   While validating results using only the training dataset, we can infer that all metrics(including f1_score) grow indefenitely to 100% as the depth of the tree increases, indicating an overfitting behaviour.\n",
    "*   For Validation and Test datasets however, model performance reaches its peak on max_depth=13, indicating that there is no more gain of information for these datasets beyond this point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Random Forest Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'n_estimators': [1, 10, 50, 100, 150, 200],\n",
    "          'max_depth': [1, 5, 10, 15, 20]}\n",
    "\n",
    "n_estimators_list = list()\n",
    "max_depth_list = list()\n",
    "\n",
    "acc_list_training = list()\n",
    "precision_list_training = list()\n",
    "recall_list_training = list()\n",
    "f1_list_training = list()\n",
    "\n",
    "acc_list_validation = list()\n",
    "precision_list_validation = list()\n",
    "recall_list_validation = list()\n",
    "f1_list_validation = list()\n",
    "\n",
    "acc_list_test = list()\n",
    "precision_list_test = list()\n",
    "recall_list_test = list()\n",
    "f1_list_test = list()\n",
    "\n",
    "\n",
    "for i in params.get('n_estimators'):\n",
    "    for j in params.get('max_depth'):\n",
    "        n_estimators_list.append(i)\n",
    "        max_depth_list.append(j)\n",
    "        #Model Definition\n",
    "        rf_model = RandomForestClassifier(n_estimators=i, max_depth=j, random_state=42)\n",
    "\n",
    "        #Model Training\n",
    "        rf_model.fit(X_training_classification, np.ravel(y_training_classification))\n",
    "\n",
    "        #Model Predict\n",
    "        yhat_rf_training    = rf_model.predict(X_training_classification)\n",
    "        yhat_rf_validation  = rf_model.predict(X_validation_classification)\n",
    "        yhat_rf_test        = rf_model.predict(X_test_classification)\n",
    "\n",
    "        #Model Scores\n",
    "        #training scores\n",
    "        acc_score_training       = mt.accuracy_score(y_training_classification, yhat_rf_training)\n",
    "        precision_score_training = mt.precision_score(y_training_classification, yhat_rf_training)\n",
    "        recall_score_training    = mt.recall_score(y_training_classification, yhat_rf_training)\n",
    "        f1_score_training        = mt.f1_score(y_training_classification, yhat_rf_training)\n",
    "    \n",
    "        acc_list_training.append(acc_score_training)\n",
    "        precision_list_training.append(precision_score_training)\n",
    "        recall_list_training.append(recall_score_training)\n",
    "        f1_list_training.append(f1_score_training)\n",
    "    \n",
    "        #validation scores\n",
    "        acc_score_validation       = mt.accuracy_score(y_validation_classification, yhat_rf_validation)\n",
    "        precision_score_validation = mt.precision_score(y_validation_classification, yhat_rf_validation)\n",
    "        recall_score_validation    = mt.recall_score(y_validation_classification, yhat_rf_validation)\n",
    "        f1_score_validation        = mt.f1_score(y_validation_classification, yhat_rf_validation)\n",
    "    \n",
    "        acc_list_validation.append(acc_score_validation)\n",
    "        precision_list_validation.append(precision_score_validation)\n",
    "        recall_list_validation.append(recall_score_validation)\n",
    "        f1_list_validation.append(f1_score_validation)\n",
    "\n",
    "        #test scores\n",
    "        acc_score_test       = mt.accuracy_score(y_test_classification, yhat_rf_test)\n",
    "        precision_score_test = mt.precision_score(y_test_classification, yhat_rf_test)\n",
    "        recall_score_test    = mt.recall_score(y_test_classification, yhat_rf_test)\n",
    "        f1_score_test        = mt.f1_score(y_test_classification, yhat_rf_test)\n",
    "    \n",
    "        acc_list_test.append(acc_score_test)\n",
    "        precision_list_test.append(precision_score_test)\n",
    "        recall_list_test.append(recall_score_test)\n",
    "        f1_list_test.append(f1_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>20</td>\n",
       "      <td>150</td>\n",
       "      <td>0.996525</td>\n",
       "      <td>0.998242</td>\n",
       "      <td>0.993732</td>\n",
       "      <td>0.995982</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    max_depth  n_estimators  accuracy_score  precision_score  recall_score  \\\n",
       "24         20           150        0.996525         0.998242      0.993732   \n",
       "\n",
       "    f1_score  \n",
       "24  0.995982  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_training_df = pd.DataFrame({'max_depth': max_depth_list, 'n_estimators':n_estimators_list,\n",
    "                               'accuracy_score':acc_list_training, 'precision_score': precision_list_training,\n",
    "                               'recall_score':recall_list_training, 'f1_score':f1_list_training})\n",
    "\n",
    "\n",
    "rf_training_max = rf_training_df.query('f1_score == f1_score.max()')\n",
    "rf_training_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>20</td>\n",
       "      <td>200</td>\n",
       "      <td>0.965185</td>\n",
       "      <td>0.974271</td>\n",
       "      <td>0.944614</td>\n",
       "      <td>0.959213</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    max_depth  n_estimators  accuracy_score  precision_score  recall_score  \\\n",
       "29         20           200        0.965185         0.974271      0.944614   \n",
       "\n",
       "    f1_score  \n",
       "29  0.959213  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_validation_df = pd.DataFrame({'max_depth': max_depth_list, 'n_estimators':n_estimators_list,\n",
    "                               'accuracy_score':acc_list_validation, 'precision_score': precision_list_validation,\n",
    "                               'recall_score':recall_list_validation, 'f1_score':f1_list_validation})\n",
    "\n",
    "rf_validation_max = rf_validation_df.query('f1_score == f1_score.max()')\n",
    "rf_validation_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>20</td>\n",
       "      <td>150</td>\n",
       "      <td>0.963928</td>\n",
       "      <td>0.971777</td>\n",
       "      <td>0.945271</td>\n",
       "      <td>0.958341</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    max_depth  n_estimators  accuracy_score  precision_score  recall_score  \\\n",
       "24         20           150        0.963928         0.971777      0.945271   \n",
       "\n",
       "    f1_score  \n",
       "24  0.958341  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_test_df = pd.DataFrame({'max_depth': max_depth_list, 'n_estimators':n_estimators_list,\n",
    "                               'accuracy_score':acc_list_test, 'precision_score': precision_list_test,\n",
    "                               'recall_score':recall_list_test, 'f1_score':f1_list_test})\n",
    "\n",
    "rf_test_max = rf_test_df.query('f1_score == f1_score.max()')\n",
    "rf_test_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Alike the Decision Tree results, a tree based model such as the Random Forest tend to overfit as you increase the number of estimators as well as tree max depth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Logistic Regression Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'C':[0.1, 0.5, 1.0, 2.0],\n",
    "          'solver': ['lbfgs', 'liblinear', 'newton-cg', 'newton-cholesky', 'sag', 'saga'],\n",
    "          'max_iter': [50, 100, 150, 200]}\n",
    "\n",
    "c_estimator_list = list()\n",
    "solver_estimator_list = list()\n",
    "max_iter_estimator_list = list()\n",
    "\n",
    "acc_list_training = list()\n",
    "precision_list_training = list()\n",
    "recall_list_training = list()\n",
    "f1_list_training = list()\n",
    "\n",
    "acc_list_validation = list()\n",
    "precision_list_validation = list()\n",
    "recall_list_validation = list()\n",
    "f1_list_validation = list()\n",
    "\n",
    "acc_list_test = list()\n",
    "precision_list_test = list()\n",
    "recall_list_test = list()\n",
    "f1_list_test = list()\n",
    "\n",
    "for i in params.get('C'):\n",
    "    for j in params.get('solver'):\n",
    "        for p in params.get('max_iter'):\n",
    "            c_estimator_list.append(i)\n",
    "            solver_estimator_list.append(j)\n",
    "            max_iter_estimator_list.append(p)\n",
    "            \n",
    "            #Model Define\n",
    "            lr_model = LogisticRegression(C=i, solver=j, max_iter=p, random_state=42)\n",
    "            \n",
    "            #Model Training\n",
    "            lr_model.fit(X_training_classification, np.ravel(y_training_classification))\n",
    "            \n",
    "            #Model Predict\n",
    "            yhat_lr_training    = lr_model.predict(X_training_classification)\n",
    "            yhat_lr_validation  = lr_model.predict(X_validation_classification)\n",
    "            yhat_lr_test        = lr_model.predict(X_test_classification)\n",
    "            \n",
    "            #Model Scores\n",
    "            #Training Scores\n",
    "            acc_score_training       = mt.accuracy_score(y_training_classification, yhat_lr_training)\n",
    "            precision_score_training = mt.precision_score(y_training_classification, yhat_lr_training)\n",
    "            recall_score_training    = mt.recall_score(y_training_classification, yhat_lr_training)\n",
    "            f1_score_training        = mt.f1_score(y_training_classification, yhat_lr_training)\n",
    "    \n",
    "            acc_list_training.append(acc_score_training)\n",
    "            precision_list_training.append(precision_score_training)\n",
    "            recall_list_training.append(recall_score_training)\n",
    "            f1_list_training.append(f1_score_training)\n",
    "            \n",
    "            #Validation Scores\n",
    "            acc_score_validation       = mt.accuracy_score(y_validation_classification, yhat_lr_validation)\n",
    "            precision_score_validation = mt.precision_score(y_validation_classification, yhat_lr_validation)\n",
    "            recall_score_validation    = mt.recall_score(y_validation_classification, yhat_lr_validation)\n",
    "            f1_score_validation        = mt.f1_score(y_validation_classification, yhat_lr_validation)\n",
    "    \n",
    "            acc_list_validation.append(acc_score_validation)\n",
    "            precision_list_validation.append(precision_score_validation)\n",
    "            recall_list_validation.append(recall_score_validation)\n",
    "            f1_list_validation.append(f1_score_validation)\n",
    "            \n",
    "            #Test Scores\n",
    "            acc_score_test       = mt.accuracy_score(y_test_classification, yhat_lr_test)\n",
    "            precision_score_test = mt.precision_score(y_test_classification, yhat_lr_test)\n",
    "            recall_score_test    = mt.recall_score(y_test_classification, yhat_lr_test)\n",
    "            f1_score_test        = mt.f1_score(y_test_classification, yhat_lr_test)\n",
    "    \n",
    "            acc_list_test.append(acc_score_test)\n",
    "            precision_list_test.append(precision_score_test)\n",
    "            recall_list_test.append(recall_score_test)\n",
    "            f1_list_test.append(f1_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>C</th>\n",
       "      <th>solver</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.1</td>\n",
       "      <td>newton-cg</td>\n",
       "      <td>100</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.1</td>\n",
       "      <td>newton-cg</td>\n",
       "      <td>150</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.1</td>\n",
       "      <td>newton-cg</td>\n",
       "      <td>200</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.1</td>\n",
       "      <td>newton-cholesky</td>\n",
       "      <td>50</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.1</td>\n",
       "      <td>newton-cholesky</td>\n",
       "      <td>100</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.1</td>\n",
       "      <td>newton-cholesky</td>\n",
       "      <td>150</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.1</td>\n",
       "      <td>newton-cholesky</td>\n",
       "      <td>200</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      C           solver  max_iter  accuracy_score  precision_score  \\\n",
       "9   0.1        newton-cg       100        0.876288         0.871866   \n",
       "10  0.1        newton-cg       150        0.876288         0.871866   \n",
       "11  0.1        newton-cg       200        0.876288         0.871866   \n",
       "12  0.1  newton-cholesky        50        0.876288         0.871866   \n",
       "13  0.1  newton-cholesky       100        0.876288         0.871866   \n",
       "14  0.1  newton-cholesky       150        0.876288         0.871866   \n",
       "15  0.1  newton-cholesky       200        0.876288         0.871866   \n",
       "\n",
       "    recall_score  f1_score  \n",
       "9       0.837661  0.854421  \n",
       "10      0.837661  0.854421  \n",
       "11      0.837661  0.854421  \n",
       "12      0.837661  0.854421  \n",
       "13      0.837661  0.854421  \n",
       "14      0.837661  0.854421  \n",
       "15      0.837661  0.854421  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_training_df = pd.DataFrame({'C': c_estimator_list, 'solver':solver_estimator_list, 'max_iter':max_iter_estimator_list,\n",
    "                               'accuracy_score':acc_list_training, 'precision_score': precision_list_training,\n",
    "                               'recall_score':recall_list_training, 'f1_score':f1_list_training})\n",
    "\n",
    "lr_training_max = lr_training_df.query('f1_score == f1_score.max()')\n",
    "lr_training_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>C</th>\n",
       "      <th>solver</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>2.0</td>\n",
       "      <td>newton-cg</td>\n",
       "      <td>100</td>\n",
       "      <td>0.874481</td>\n",
       "      <td>0.869421</td>\n",
       "      <td>0.83592</td>\n",
       "      <td>0.852341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>2.0</td>\n",
       "      <td>newton-cg</td>\n",
       "      <td>150</td>\n",
       "      <td>0.874481</td>\n",
       "      <td>0.869421</td>\n",
       "      <td>0.83592</td>\n",
       "      <td>0.852341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>2.0</td>\n",
       "      <td>newton-cg</td>\n",
       "      <td>200</td>\n",
       "      <td>0.874481</td>\n",
       "      <td>0.869421</td>\n",
       "      <td>0.83592</td>\n",
       "      <td>0.852341</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      C     solver  max_iter  accuracy_score  precision_score  recall_score  \\\n",
       "81  2.0  newton-cg       100        0.874481         0.869421       0.83592   \n",
       "82  2.0  newton-cg       150        0.874481         0.869421       0.83592   \n",
       "83  2.0  newton-cg       200        0.874481         0.869421       0.83592   \n",
       "\n",
       "    f1_score  \n",
       "81  0.852341  \n",
       "82  0.852341  \n",
       "83  0.852341  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_validation_df = pd.DataFrame({'C': c_estimator_list, 'solver':solver_estimator_list, 'max_iter':max_iter_estimator_list,\n",
    "                               'accuracy_score':acc_list_validation, 'precision_score': precision_list_validation,\n",
    "                               'recall_score':recall_list_validation, 'f1_score':f1_list_validation})\n",
    "\n",
    "lr_validation_max = lr_validation_df.query('f1_score == f1_score.max()')\n",
    "lr_validation_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>C</th>\n",
       "      <th>solver</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>0.5</td>\n",
       "      <td>newton-cg</td>\n",
       "      <td>50</td>\n",
       "      <td>0.871857</td>\n",
       "      <td>0.868014</td>\n",
       "      <td>0.83502</td>\n",
       "      <td>0.851197</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      C     solver  max_iter  accuracy_score  precision_score  recall_score  \\\n",
       "32  0.5  newton-cg        50        0.871857         0.868014       0.83502   \n",
       "\n",
       "    f1_score  \n",
       "32  0.851197  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_test_df = pd.DataFrame({'C': c_estimator_list, 'solver':solver_estimator_list, 'max_iter':max_iter_estimator_list,\n",
    "                               'accuracy_score':acc_list_test, 'precision_score': precision_list_test,\n",
    "                               'recall_score':recall_list_test, 'f1_score':f1_list_test})\n",
    "\n",
    "lr_test_max = lr_test_df.query('f1_score == f1_score.max()')\n",
    "lr_test_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   It was observed that for these datasets, the optimization algorithms that showed best performance were the ones that contain L2 penalty support. (newton-cg and newton-cholesky)\n",
    "*   Other methods (lbfgs, sag, liblinear) either failed to converge due to not approximating the cost function well or by applying a not smooth penalty(L1) to relatively small datasets(maximum 73k rows).\n",
    "*   Both C and max_iter parameters proved to be secundary upon cost function convergence, showing in comparison little influence on final metric results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.5 Final Results for Classification Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5.1 Training Dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn_training_max = knn_training_max.drop('neighbors', axis=1)\n",
    "knn_training_max['model'] = 'KNN'\n",
    "knn_training_max = knn_training_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "tree_training_max = tree_training_max.drop('max_depth', axis=1)\n",
    "tree_training_max['model'] = 'Decision Tree'\n",
    "tree_training_max = tree_training_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "rf_training_max = rf_training_max.drop(['n_estimators', 'max_depth'], axis=1)\n",
    "rf_training_max['model'] = 'Random Forest'\n",
    "rf_training_max = rf_training_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "lr_training_max = lr_training_max.drop(['C', 'solver', 'max_iter'], axis=1)\n",
    "lr_training_max['model'] = 'Logistic Regression'\n",
    "lr_training_max = lr_training_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>KNN</td>\n",
       "      <td>0.832186</td>\n",
       "      <td>0.812008</td>\n",
       "      <td>0.797410</td>\n",
       "      <td>0.804643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Decision Tree</td>\n",
       "      <td>0.989712</td>\n",
       "      <td>0.993089</td>\n",
       "      <td>0.983104</td>\n",
       "      <td>0.988072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.996525</td>\n",
       "      <td>0.998242</td>\n",
       "      <td>0.993732</td>\n",
       "      <td>0.995982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.876288</td>\n",
       "      <td>0.871866</td>\n",
       "      <td>0.837661</td>\n",
       "      <td>0.854421</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  model  accuracy_score  precision_score  recall_score  \\\n",
       "0                   KNN        0.832186         0.812008      0.797410   \n",
       "9         Decision Tree        0.989712         0.993089      0.983104   \n",
       "24        Random Forest        0.996525         0.998242      0.993732   \n",
       "9   Logistic Regression        0.876288         0.871866      0.837661   \n",
       "\n",
       "    f1_score  \n",
       "0   0.804643  \n",
       "9   0.988072  \n",
       "24  0.995982  \n",
       "9   0.854421  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_classification_results = pd.concat([knn_training_max, tree_training_max, rf_training_max, lr_training_max.iloc[[0]]])\n",
    "training_classification_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5.2 Validation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn_validation_max = knn_validation_max.drop('neighbors', axis=1)\n",
    "knn_validation_max['model'] = 'KNN'\n",
    "knn_validation_max = knn_validation_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "tree_validation_max = tree_validation_max.drop('max_depth', axis=1)\n",
    "tree_validation_max['model'] = 'Decision Tree'\n",
    "tree_validation_max = tree_validation_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "rf_validation_max = rf_validation_max.drop(['n_estimators', 'max_depth'], axis=1)\n",
    "rf_validation_max['model'] = 'Random Forest'\n",
    "rf_validation_max = rf_validation_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "lr_validation_max = lr_validation_max.drop(['C', 'solver', 'max_iter'], axis=1)\n",
    "lr_validation_max['model'] = 'Logistic Regression'\n",
    "lr_validation_max = lr_validation_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>KNN</td>\n",
       "      <td>0.676277</td>\n",
       "      <td>0.627851</td>\n",
       "      <td>0.621278</td>\n",
       "      <td>0.624548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Decision Tree</td>\n",
       "      <td>0.952315</td>\n",
       "      <td>0.956300</td>\n",
       "      <td>0.932586</td>\n",
       "      <td>0.944294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.965185</td>\n",
       "      <td>0.974271</td>\n",
       "      <td>0.944614</td>\n",
       "      <td>0.959213</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.874481</td>\n",
       "      <td>0.869421</td>\n",
       "      <td>0.835920</td>\n",
       "      <td>0.852341</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  model  accuracy_score  precision_score  recall_score  \\\n",
       "0                   KNN        0.676277         0.627851      0.621278   \n",
       "6         Decision Tree        0.952315         0.956300      0.932586   \n",
       "29        Random Forest        0.965185         0.974271      0.944614   \n",
       "81  Logistic Regression        0.874481         0.869421      0.835920   \n",
       "\n",
       "    f1_score  \n",
       "0   0.624548  \n",
       "6   0.944294  \n",
       "29  0.959213  \n",
       "81  0.852341  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_classification_results = pd.concat([knn_validation_max, tree_validation_max, rf_validation_max, lr_validation_max.iloc[[0]]])\n",
    "validation_classification_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5.3 Test Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn_test_max = knn_test_max.drop('neighbors', axis=1)\n",
    "knn_test_max['model'] = 'KNN'\n",
    "knn_test_max = knn_test_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "tree_test_max = tree_test_max.drop('max_depth', axis=1)\n",
    "tree_test_max['model'] = 'Decision Tree'\n",
    "tree_test_max = tree_test_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "rf_test_max = rf_test_max.drop(['n_estimators', 'max_depth'], axis=1)\n",
    "rf_test_max['model'] = 'Random Forest'\n",
    "rf_test_max = rf_test_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]\n",
    "\n",
    "lr_test_max = lr_test_max.drop(['C', 'solver', 'max_iter'], axis=1)\n",
    "lr_test_max['model'] = 'Logistic Regression'\n",
    "lr_test_max = lr_test_max[['model', 'accuracy_score', 'precision_score', 'recall_score', 'f1_score']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>accuracy_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>KNN</td>\n",
       "      <td>0.672228</td>\n",
       "      <td>0.630462</td>\n",
       "      <td>0.611879</td>\n",
       "      <td>0.621031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Decision Tree</td>\n",
       "      <td>0.951802</td>\n",
       "      <td>0.953881</td>\n",
       "      <td>0.935416</td>\n",
       "      <td>0.944558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.963928</td>\n",
       "      <td>0.971777</td>\n",
       "      <td>0.945271</td>\n",
       "      <td>0.958341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.871857</td>\n",
       "      <td>0.868014</td>\n",
       "      <td>0.835020</td>\n",
       "      <td>0.851197</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  model  accuracy_score  precision_score  recall_score  \\\n",
       "0                   KNN        0.672228         0.630462      0.611879   \n",
       "6         Decision Tree        0.951802         0.953881      0.935416   \n",
       "24        Random Forest        0.963928         0.971777      0.945271   \n",
       "32  Logistic Regression        0.871857         0.868014      0.835020   \n",
       "\n",
       "    f1_score  \n",
       "0   0.621031  \n",
       "6   0.944558  \n",
       "24  0.958341  \n",
       "32  0.851197  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_classification_results = pd.concat([knn_test_max, tree_test_max, rf_test_max, lr_test_max.iloc[[0]]])\n",
    "test_classification_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.0 REGRESSION MODELS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   The main metric used for evaluation on overall model performance will be the MAPE (Mean Absolute Percentual Error), since it expresses the algorithm performance alongside the proportion of the error it presents, standing as an easier way to quantify the error obtained by each model whether than using its absolute values.\n",
    "*   Other error metrics will also be listed(Mean Squared Error(MSE), Rooted Mean Squared Error(RMSE), Mean Absolute Error(MSE), R2 Score), as well as the hyperparameter set tunned for each iteration, in order to obtain insights on how each model works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_training_regression = pd.read_csv('datasets\\\\regression\\\\X_training.csv')\n",
    "y_training_regression = pd.read_csv('datasets\\\\regression\\\\y_training.csv')\n",
    "\n",
    "X_validation_regression = pd.read_csv('datasets\\\\regression\\\\X_validation.csv')\n",
    "y_validation_regression = pd.read_csv('datasets\\\\regression\\\\y_validation.csv')\n",
    "\n",
    "X_test_regression = pd.read_csv('datasets\\\\regression\\\\X_test.csv')\n",
    "y_test_regression = pd.read_csv('datasets\\\\regression\\\\y_test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training dataset: rows = 10547, columns = 13\n",
      "Validation dataset: rows = 4521, columns = 13\n",
      "Test dataset: rows = 3767, columns = 13\n",
      "Number of classes: 101\n"
     ]
    }
   ],
   "source": [
    "print(f'Training dataset: rows = {X_training_regression.shape[0]}, columns = {X_training_regression.shape[1]}')\n",
    "print(f'Validation dataset: rows = {X_validation_regression.shape[0]}, columns = {X_validation_regression.shape[1]}')\n",
    "print(f'Test dataset: rows = {X_test_regression.shape[0]}, columns = {X_test_regression.shape[1]}')\n",
    "\n",
    "print(f'Number of classes: {len(np.unique(y_training_regression))}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Every class variable array ('y' values) match sizes with their features dataset counterparts ('x' variables)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Model Definition\n",
    "linear_regression_model = LinearRegression()\n",
    "    \n",
    "#Model Train\n",
    "linear_regression_model.fit(X_training_regression, y_training_regression)\n",
    "    \n",
    "#Model Predict\n",
    "yhat_linear_regression_training   = linear_regression_model.predict(X_training_regression)\n",
    "yhat_linear_regression_validation = linear_regression_model.predict(X_validation_regression)\n",
    "yhat_linear_regression_test       = linear_regression_model.predict(X_test_regression)\n",
    "    \n",
    "#training scores\n",
    "r2_score_training       = mt.r2_score(y_training_regression, yhat_linear_regression_training)\n",
    "mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_linear_regression_training)\n",
    "rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_linear_regression_training))\n",
    "mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_linear_regression_training)\n",
    "mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_linear_regression_training)\n",
    "    \n",
    "#validation scores\n",
    "r2_score_validation       = mt.r2_score(y_validation_regression, yhat_linear_regression_validation)\n",
    "mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_linear_regression_validation)\n",
    "rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_linear_regression_validation))\n",
    "mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_linear_regression_validation)\n",
    "mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_linear_regression_validation)\n",
    "    \n",
    "#test scores\n",
    "r2_score_test       = mt.r2_score(y_test_regression, yhat_linear_regression_test)\n",
    "mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_linear_regression_test)\n",
    "rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_linear_regression_test))\n",
    "mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_linear_regression_test)\n",
    "mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_linear_regression_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996112</td>\n",
       "      <td>21.354065</td>\n",
       "      <td>16.998249</td>\n",
       "      <td>8.653186</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0  0.046058  455.996112   21.354065  16.998249    8.653186"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linear_regression_training_min = pd.DataFrame({'r2_score': r2_score_training, 'mse_score': mse_score_training,\n",
    "                                               'rmse_score': rmse_score_training, 'mae_score': mae_score_training,\n",
    "                                               'mape_score': mape_score_training}, index= [0])\n",
    "linear_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.039925</td>\n",
       "      <td>458.447042</td>\n",
       "      <td>21.411376</td>\n",
       "      <td>17.039754</td>\n",
       "      <td>8.682542</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0  0.039925  458.447042   21.411376  17.039754    8.682542"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linear_regression_validation_min = pd.DataFrame({'r2_score': r2_score_validation, 'mse_score': mse_score_validation,\n",
    "                                               'rmse_score': rmse_score_validation, 'mae_score': mae_score_validation,\n",
    "                                               'mape_score': mape_score_validation}, index= [0])\n",
    "linear_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.052317</td>\n",
       "      <td>461.427719</td>\n",
       "      <td>21.480869</td>\n",
       "      <td>17.129965</td>\n",
       "      <td>8.521859</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0  0.052317  461.427719   21.480869  17.129965    8.521859"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linear_regression_test_min = pd.DataFrame({'r2_score': r2_score_test, 'mse_score': mse_score_test,\n",
    "                                               'rmse_score': rmse_score_test, 'mae_score': mae_score_test,\n",
    "                                               'mape_score': mape_score_test}, index= [0])\n",
    "linear_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   The Linear Regression model serves as a base model for further comparison, as it is conceptually the simplest of all regression models.\n",
    "*   Given the simple nature of a linear model, there are no parameters to iterate for its construction.\n",
    "*   The value obtained for r2_score (0.0 value) indicates that data is sparse, which means that a linear model wont fit well to how this dataset behaves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Linear Regression Lasso (L1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'alpha': [1, 50, 100, 200],\n",
    "          'max_iter': [1, 50, 100, 200]}\n",
    "\n",
    "alpha_estimator_list = list()\n",
    "max_iter_estimator_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('alpha'):\n",
    "    for j in params.get('max_iter'):\n",
    "        alpha_estimator_list.append(i)\n",
    "        max_iter_estimator_list.append(j)\n",
    "        #Model Define\n",
    "        lasso_regression = Lasso(alpha=i, max_iter=j, random_state=42)\n",
    "    \n",
    "        #Model Training\n",
    "        lasso_regression.fit(X_training_regression, np.ravel(y_training_regression))\n",
    "    \n",
    "        #Model Predict\n",
    "        yhat_lasso_regression_training   = lasso_regression.predict(X_training_regression)\n",
    "        yhat_lasso_regression_validation = lasso_regression.predict(X_validation_regression)\n",
    "        yhat_lasso_regression_test       = lasso_regression.predict(X_test_regression)\n",
    "    \n",
    "        #training scores\n",
    "        r2_score_training       = mt.r2_score(y_training_regression, yhat_lasso_regression_training)\n",
    "        mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_lasso_regression_training)\n",
    "        rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_lasso_regression_training))\n",
    "        mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_lasso_regression_training)\n",
    "        mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_lasso_regression_training)\n",
    "    \n",
    "        r2_list_training.append(r2_score_training)\n",
    "        mse_list_training.append(mse_score_training)\n",
    "        rmse_list_training.append(rmse_score_training)\n",
    "        mae_list_training.append(mae_score_training)\n",
    "        mape_list_training.append(mape_score_training)\n",
    "    \n",
    "        #validation scores\n",
    "        r2_score_validation       = mt.r2_score(y_validation_regression, yhat_lasso_regression_validation)\n",
    "        mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_lasso_regression_validation)\n",
    "        rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_lasso_regression_validation))\n",
    "        mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_lasso_regression_validation)\n",
    "        mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_lasso_regression_validation)\n",
    "    \n",
    "        r2_list_validation.append(r2_score_validation)\n",
    "        mse_list_validation.append(mse_score_validation)\n",
    "        rmse_list_validation.append(rmse_score_validation)\n",
    "        mae_list_validation.append(mae_score_validation)\n",
    "        mape_list_validation.append(mape_score_validation)\n",
    "        \n",
    "        #test scores\n",
    "        r2_score_test       = mt.r2_score(y_test_regression, yhat_lasso_regression_test)\n",
    "        mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_lasso_regression_test)\n",
    "        rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_lasso_regression_test))\n",
    "        mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_lasso_regression_test)\n",
    "        mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_lasso_regression_test)\n",
    "    \n",
    "        r2_list_test.append(r2_score_test)\n",
    "        mse_list_test.append(mse_score_test)\n",
    "        rmse_list_test.append(rmse_score_test)\n",
    "        mae_list_test.append(mae_score_test)\n",
    "        mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.007401</td>\n",
       "      <td>474.474834</td>\n",
       "      <td>21.782443</td>\n",
       "      <td>17.305484</td>\n",
       "      <td>8.736697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>0.007401</td>\n",
       "      <td>474.474834</td>\n",
       "      <td>21.782443</td>\n",
       "      <td>17.305484</td>\n",
       "      <td>8.736697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>0.007401</td>\n",
       "      <td>474.474834</td>\n",
       "      <td>21.782443</td>\n",
       "      <td>17.305484</td>\n",
       "      <td>8.736697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>0.007401</td>\n",
       "      <td>474.474834</td>\n",
       "      <td>21.782443</td>\n",
       "      <td>17.305484</td>\n",
       "      <td>8.736697</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0      1         1  0.007401  474.474834   21.782443  17.305484    8.736697\n",
       "1      1        50  0.007401  474.474834   21.782443  17.305484    8.736697\n",
       "2      1       100  0.007401  474.474834   21.782443  17.305484    8.736697\n",
       "3      1       200  0.007401  474.474834   21.782443  17.305484    8.736697"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_regression_training_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'max_iter': max_iter_estimator_list,\n",
    "                                              'r2_score': r2_list_training, 'mse_score': mse_list_training,\n",
    "                                              'rmse_score': rmse_list_training, 'mae_score': mae_list_training,\n",
    "                                              'mape_score': mape_list_training})\n",
    "\n",
    "\n",
    "lasso_regression_training_min = lasso_regression_training_df.query('mape_score == mape_score.min()')\n",
    "lasso_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>50</td>\n",
       "      <td>50</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>50</td>\n",
       "      <td>100</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>50</td>\n",
       "      <td>200</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>100</td>\n",
       "      <td>50</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>100</td>\n",
       "      <td>200</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>200</td>\n",
       "      <td>1</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>200</td>\n",
       "      <td>50</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>200</td>\n",
       "      <td>100</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>200</td>\n",
       "      <td>200</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    alpha  max_iter      r2_score   mse_score  rmse_score  mae_score  \\\n",
       "4      50         1 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "5      50        50 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "6      50       100 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "7      50       200 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "8     100         1 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "9     100        50 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "10    100       100 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "11    100       200 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "12    200         1 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "13    200        50 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "14    200       100 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "15    200       200 -7.197077e-07  477.511956   21.852047  17.352836   \n",
       "\n",
       "    mape_score  \n",
       "4     8.678722  \n",
       "5     8.678722  \n",
       "6     8.678722  \n",
       "7     8.678722  \n",
       "8     8.678722  \n",
       "9     8.678722  \n",
       "10    8.678722  \n",
       "11    8.678722  \n",
       "12    8.678722  \n",
       "13    8.678722  \n",
       "14    8.678722  \n",
       "15    8.678722  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_regression_validation_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'max_iter': max_iter_estimator_list,\n",
    "                                              'r2_score': r2_list_validation, 'mse_score': mse_list_validation,\n",
    "                                              'rmse_score': rmse_list_validation, 'mae_score': mae_list_validation,\n",
    "                                              'mape_score': mape_list_validation})\n",
    "\n",
    "lasso_regression_validation_min = lasso_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "lasso_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>50</td>\n",
       "      <td>50</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>50</td>\n",
       "      <td>100</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>50</td>\n",
       "      <td>200</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>100</td>\n",
       "      <td>50</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>100</td>\n",
       "      <td>200</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>200</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>200</td>\n",
       "      <td>50</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>200</td>\n",
       "      <td>100</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>200</td>\n",
       "      <td>200</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "4      50         1 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "5      50        50 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "6      50       100 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "7      50       200 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "8     100         1 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "9     100        50 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "10    100       100 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "11    100       200 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "12    200         1 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "13    200        50 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "14    200       100 -0.000124  486.961469   22.067203  17.551492     8.71455\n",
       "15    200       200 -0.000124  486.961469   22.067203  17.551492     8.71455"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_regression_test_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'max_iter': max_iter_estimator_list,\n",
    "                                              'r2_score': r2_list_test, 'mse_score': mse_list_test,\n",
    "                                              'rmse_score': rmse_list_test, 'mae_score': mae_list_test,\n",
    "                                              'mape_score': mape_list_test})\n",
    "\n",
    "\n",
    "lasso_regression_test_min = lasso_regression_test_df.query('mape_score == mape_score.min()')\n",
    "lasso_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Higher alpha values tend to aproximate r2_score results to 0 as aspected; Alpha is a parameter that multiplies with L1 penalty factor our regression cost function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Linear Regression Ridge (L2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'alpha': [1, 50, 100, 200],\n",
    "          'max_iter': [1, 50, 100, 200]}\n",
    "\n",
    "alpha_estimator_list = list()\n",
    "max_iter_estimator_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('alpha'):\n",
    "    for j in params.get('max_iter'):\n",
    "        alpha_estimator_list.append(i)\n",
    "        max_iter_estimator_list.append(j)\n",
    "        #Model Define\n",
    "        ridge_regression = Ridge(alpha=i, max_iter=j, random_state=42)\n",
    "    \n",
    "        #Model Training\n",
    "        ridge_regression.fit(X_training_regression, np.ravel(y_training_regression))\n",
    "    \n",
    "        #Model Predict\n",
    "        yhat_ridge_regression_training   = ridge_regression.predict(X_training_regression)\n",
    "        yhat_ridge_regression_validation = ridge_regression.predict(X_validation_regression)\n",
    "        yhat_ridge_regression_test       = ridge_regression.predict(X_test_regression)\n",
    "    \n",
    "        #training scores\n",
    "        r2_score_training       = mt.r2_score(y_training_regression, yhat_ridge_regression_training)\n",
    "        mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_ridge_regression_training)\n",
    "        rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_ridge_regression_training))\n",
    "        mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_ridge_regression_training)\n",
    "        mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_ridge_regression_training)\n",
    "    \n",
    "        r2_list_training.append(r2_score_training)\n",
    "        mse_list_training.append(mse_score_training)\n",
    "        rmse_list_training.append(rmse_score_training)\n",
    "        mae_list_training.append(mae_score_training)\n",
    "        mape_list_training.append(mape_score_training)\n",
    "    \n",
    "        #validation scores\n",
    "        r2_score_validation       = mt.r2_score(y_validation_regression, yhat_ridge_regression_validation)\n",
    "        mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_ridge_regression_validation)\n",
    "        rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_ridge_regression_validation))\n",
    "        mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_ridge_regression_validation)\n",
    "        mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_ridge_regression_validation)\n",
    "    \n",
    "        r2_list_validation.append(r2_score_validation)\n",
    "        mse_list_validation.append(mse_score_validation)\n",
    "        rmse_list_validation.append(rmse_score_validation)\n",
    "        mae_list_validation.append(mae_score_validation)\n",
    "        mape_list_validation.append(mape_score_validation)\n",
    "        \n",
    "        #test scores\n",
    "        r2_score_test       = mt.r2_score(y_test_regression, yhat_ridge_regression_test)\n",
    "        mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_ridge_regression_test)\n",
    "        rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_ridge_regression_test))\n",
    "        mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_ridge_regression_test)\n",
    "        mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_ridge_regression_test)\n",
    "    \n",
    "        r2_list_test.append(r2_score_test)\n",
    "        mse_list_test.append(mse_score_test)\n",
    "        rmse_list_test.append(rmse_score_test)\n",
    "        mae_list_test.append(mae_score_test)\n",
    "        mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996401</td>\n",
       "      <td>21.354072</td>\n",
       "      <td>16.998308</td>\n",
       "      <td>8.653415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996401</td>\n",
       "      <td>21.354072</td>\n",
       "      <td>16.998308</td>\n",
       "      <td>8.653415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996401</td>\n",
       "      <td>21.354072</td>\n",
       "      <td>16.998308</td>\n",
       "      <td>8.653415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996401</td>\n",
       "      <td>21.354072</td>\n",
       "      <td>16.998308</td>\n",
       "      <td>8.653415</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0      1         1  0.046058  455.996401   21.354072  16.998308    8.653415\n",
       "1      1        50  0.046058  455.996401   21.354072  16.998308    8.653415\n",
       "2      1       100  0.046058  455.996401   21.354072  16.998308    8.653415\n",
       "3      1       200  0.046058  455.996401   21.354072  16.998308    8.653415"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_regression_training_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'max_iter': max_iter_estimator_list,\n",
    "                                              'r2_score': r2_list_training, 'mse_score': mse_list_training,\n",
    "                                              'rmse_score': rmse_list_training, 'mae_score': mae_list_training,\n",
    "                                              'mape_score': mape_list_training})\n",
    "\n",
    "ridge_regression_training_min = ridge_regression_training_df.query('mape_score == mape_score.min()')\n",
    "ridge_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>200</td>\n",
       "      <td>1</td>\n",
       "      <td>0.03681</td>\n",
       "      <td>459.934208</td>\n",
       "      <td>21.446077</td>\n",
       "      <td>17.043389</td>\n",
       "      <td>8.672408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>200</td>\n",
       "      <td>50</td>\n",
       "      <td>0.03681</td>\n",
       "      <td>459.934208</td>\n",
       "      <td>21.446077</td>\n",
       "      <td>17.043389</td>\n",
       "      <td>8.672408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>200</td>\n",
       "      <td>100</td>\n",
       "      <td>0.03681</td>\n",
       "      <td>459.934208</td>\n",
       "      <td>21.446077</td>\n",
       "      <td>17.043389</td>\n",
       "      <td>8.672408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>200</td>\n",
       "      <td>200</td>\n",
       "      <td>0.03681</td>\n",
       "      <td>459.934208</td>\n",
       "      <td>21.446077</td>\n",
       "      <td>17.043389</td>\n",
       "      <td>8.672408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "12    200         1   0.03681  459.934208   21.446077  17.043389    8.672408\n",
       "13    200        50   0.03681  459.934208   21.446077  17.043389    8.672408\n",
       "14    200       100   0.03681  459.934208   21.446077  17.043389    8.672408\n",
       "15    200       200   0.03681  459.934208   21.446077  17.043389    8.672408"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_regression_validation_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'max_iter': max_iter_estimator_list,\n",
    "                                              'r2_score': r2_list_validation, 'mse_score': mse_list_validation,\n",
    "                                              'rmse_score': rmse_list_validation, 'mae_score': mae_list_validation,\n",
    "                                              'mape_score': mape_list_validation})\n",
    "\n",
    "ridge_regression_validation_min = ridge_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "ridge_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.05231</td>\n",
       "      <td>461.431102</td>\n",
       "      <td>21.480947</td>\n",
       "      <td>17.129678</td>\n",
       "      <td>8.522815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>0.05231</td>\n",
       "      <td>461.431102</td>\n",
       "      <td>21.480947</td>\n",
       "      <td>17.129678</td>\n",
       "      <td>8.522815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>0.05231</td>\n",
       "      <td>461.431102</td>\n",
       "      <td>21.480947</td>\n",
       "      <td>17.129678</td>\n",
       "      <td>8.522815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>0.05231</td>\n",
       "      <td>461.431102</td>\n",
       "      <td>21.480947</td>\n",
       "      <td>17.129678</td>\n",
       "      <td>8.522815</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0      1         1   0.05231  461.431102   21.480947  17.129678    8.522815\n",
       "1      1        50   0.05231  461.431102   21.480947  17.129678    8.522815\n",
       "2      1       100   0.05231  461.431102   21.480947  17.129678    8.522815\n",
       "3      1       200   0.05231  461.431102   21.480947  17.129678    8.522815"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_regression_test_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'max_iter': max_iter_estimator_list,\n",
    "                                              'r2_score': r2_list_test, 'mse_score': mse_list_test,\n",
    "                                              'rmse_score': rmse_list_test, 'mae_score': mae_list_test,\n",
    "                                              'mape_score': mape_list_test})\n",
    "\n",
    "ridge_regression_test_min = ridge_regression_test_df.query('mape_score == mape_score.min()')\n",
    "ridge_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   As alpha here is a multiplier to the penalty factor L2 for ridge regression, its influence on r2_score is more subtle, as we can see values dont tend abruptly to 0 anymore in comparison to Lasso Regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Linear Regression Elastic Net (L1 & L2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'alpha': [1, 50, 100, 200],\n",
    "          'l1_ratio': [0, 0.2, 0.5, 0.7, 1.0],\n",
    "          'max_iter': [1, 50, 100, 200]}\n",
    "\n",
    "alpha_estimator_list = list()\n",
    "l1_ratio_estimator_list = list()\n",
    "max_iter_estimator_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('alpha'):\n",
    "    for j in params.get('l1_ratio'):\n",
    "        for p in params.get('max_iter'):    \n",
    "            alpha_estimator_list.append(i)\n",
    "            l1_ratio_estimator_list.append(j)\n",
    "            max_iter_estimator_list.append(p)\n",
    "            #Model Define\n",
    "            elastic_net = ElasticNet(alpha=i, l1_ratio=j, max_iter=p, random_state=42)\n",
    "    \n",
    "            #Model Training\n",
    "            elastic_net.fit(X_training_regression, np.ravel(y_training_regression))\n",
    "    \n",
    "            #Model Predict\n",
    "            yhat_elastic_net_training   = elastic_net.predict(X_training_regression)\n",
    "            yhat_elastic_net_validation = elastic_net.predict(X_validation_regression)\n",
    "            yhat_elastic_net_test       = elastic_net.predict(X_test_regression)\n",
    "    \n",
    "            #training scores\n",
    "            r2_score_training       = mt.r2_score(y_training_regression, yhat_elastic_net_training)\n",
    "            mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_elastic_net_training)\n",
    "            rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_elastic_net_training))\n",
    "            mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_elastic_net_training)\n",
    "            mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_elastic_net_training)\n",
    "    \n",
    "            r2_list_training.append(r2_score_training)\n",
    "            mse_list_training.append(mse_score_training)\n",
    "            rmse_list_training.append(rmse_score_training)\n",
    "            mae_list_training.append(mae_score_training)\n",
    "            mape_list_training.append(mape_score_training)\n",
    "    \n",
    "            #validation scores\n",
    "            r2_score_validation       = mt.r2_score(y_validation_regression, yhat_elastic_net_validation)\n",
    "            mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_elastic_net_validation)\n",
    "            rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_elastic_net_validation))\n",
    "            mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_elastic_net_validation)\n",
    "            mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_elastic_net_validation)\n",
    "    \n",
    "            r2_list_validation.append(r2_score_validation)\n",
    "            mse_list_validation.append(mse_score_validation)\n",
    "            rmse_list_validation.append(rmse_score_validation)\n",
    "            mae_list_validation.append(mae_score_validation)\n",
    "            mape_list_validation.append(mape_score_validation)\n",
    "        \n",
    "            #test scores\n",
    "            r2_score_test       = mt.r2_score(y_test_regression, yhat_elastic_net_test)\n",
    "            mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_elastic_net_test)\n",
    "            rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_elastic_net_test))\n",
    "            mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_elastic_net_test)\n",
    "            mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_elastic_net_test)\n",
    "    \n",
    "            r2_list_test.append(r2_score_test)\n",
    "            mse_list_test.append(mse_score_test)\n",
    "            rmse_list_test.append(rmse_score_test)\n",
    "            mae_list_test.append(mae_score_test)\n",
    "            mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>l1_ratio</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50</td>\n",
       "      <td>0.010715</td>\n",
       "      <td>472.890766</td>\n",
       "      <td>21.746052</td>\n",
       "      <td>17.277098</td>\n",
       "      <td>8.71988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100</td>\n",
       "      <td>0.010715</td>\n",
       "      <td>472.890766</td>\n",
       "      <td>21.746052</td>\n",
       "      <td>17.277098</td>\n",
       "      <td>8.71988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>200</td>\n",
       "      <td>0.010715</td>\n",
       "      <td>472.890766</td>\n",
       "      <td>21.746052</td>\n",
       "      <td>17.277098</td>\n",
       "      <td>8.71988</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   alpha  l1_ratio  max_iter  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "1      1       0.0        50  0.010715  472.890766   21.746052  17.277098   \n",
       "2      1       0.0       100  0.010715  472.890766   21.746052  17.277098   \n",
       "3      1       0.0       200  0.010715  472.890766   21.746052  17.277098   \n",
       "\n",
       "   mape_score  \n",
       "1     8.71988  \n",
       "2     8.71988  \n",
       "3     8.71988  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elastic_net_training_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'l1_ratio': l1_ratio_estimator_list,\n",
    "                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_training, \n",
    "                                         'mse_score': mse_list_training, 'rmse_score': rmse_list_training, \n",
    "                                         'mae_score': mae_list_training, 'mape_score': mape_list_training})\n",
    "\n",
    "elastic_net_training_min = elastic_net_training_df.query('mape_score == mape_score.min()')\n",
    "elastic_net_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>l1_ratio</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>50</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>100</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>200</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>50</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    alpha  l1_ratio  max_iter      r2_score   mse_score  rmse_score  \\\n",
       "24     50       0.2         1 -7.197077e-07  477.511956   21.852047   \n",
       "25     50       0.2        50 -7.197077e-07  477.511956   21.852047   \n",
       "26     50       0.2       100 -7.197077e-07  477.511956   21.852047   \n",
       "27     50       0.2       200 -7.197077e-07  477.511956   21.852047   \n",
       "28     50       0.5         1 -7.197077e-07  477.511956   21.852047   \n",
       "\n",
       "    mae_score  mape_score  \n",
       "24  17.352836    8.678722  \n",
       "25  17.352836    8.678722  \n",
       "26  17.352836    8.678722  \n",
       "27  17.352836    8.678722  \n",
       "28  17.352836    8.678722  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elastic_net_validation_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'l1_ratio': l1_ratio_estimator_list,\n",
    "                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_validation, \n",
    "                                         'mse_score': mse_list_validation, 'rmse_score': rmse_list_validation, \n",
    "                                         'mae_score': mae_list_validation, 'mape_score': mape_list_validation})\n",
    "\n",
    "elastic_net_validation_min = elastic_net_validation_df.query('mape_score == mape_score.min()')\n",
    "elastic_net_validation_min.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>l1_ratio</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>50</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>100</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>50</td>\n",
       "      <td>0.2</td>\n",
       "      <td>200</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>50</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.71455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    alpha  l1_ratio  max_iter  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "24     50       0.2         1 -0.000124  486.961469   22.067203  17.551492   \n",
       "25     50       0.2        50 -0.000124  486.961469   22.067203  17.551492   \n",
       "26     50       0.2       100 -0.000124  486.961469   22.067203  17.551492   \n",
       "27     50       0.2       200 -0.000124  486.961469   22.067203  17.551492   \n",
       "28     50       0.5         1 -0.000124  486.961469   22.067203  17.551492   \n",
       "\n",
       "    mape_score  \n",
       "24     8.71455  \n",
       "25     8.71455  \n",
       "26     8.71455  \n",
       "27     8.71455  \n",
       "28     8.71455  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elastic_net_test_df = pd.DataFrame({ 'alpha': alpha_estimator_list, 'l1_ratio': l1_ratio_estimator_list,\n",
    "                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_test, \n",
    "                                         'mse_score': mse_list_test, 'rmse_score': rmse_list_test, \n",
    "                                         'mae_score': mae_list_test, 'mape_score': mape_list_test})\n",
    "\n",
    "elastic_net_test_min = elastic_net_test_df.query('mape_score == mape_score.min()')\n",
    "elastic_net_test_min.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Elastic Net models have both L1 and L2 penalty factors combined in its regression cost function optimization. \n",
    "    'l1_ratio' parameter is then a mixing parameter: For 'l1_ratio=0', The Penalty is exclusively L2; for 'l1_ratio=1', the penalty is exclusively L1.\n",
    "*   We can see that results pointed better error optimization (mape minimization) when l1_ratio geared towards values closer to 0, which is expected since L1 penalty is a more severe regularization factor in comparison to L2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.5 Polinomial Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'degree': np.arange(1, 6).tolist()}\n",
    "\n",
    "degree_estimator_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('degree'):\n",
    "    degree_estimator_list.append(i)\n",
    "    #Model Define\n",
    "    polynomial = pp.PolynomialFeatures(degree=i)\n",
    "    X_poly_training = polynomial.fit_transform(X_training_regression)\n",
    "    X_poly_validation = polynomial.fit_transform(X_validation_regression)\n",
    "    X_poly_test = polynomial.fit_transform(X_test_regression)\n",
    "    \n",
    "    polynomial_linear_regression = LinearRegression()\n",
    "    \n",
    "    #Model Training\n",
    "    polynomial_linear_regression.fit(X_poly_training, np.ravel(y_training_regression))\n",
    "    \n",
    "    #Model Predict\n",
    "    yhat_polynomial_lr_training   = polynomial_linear_regression.predict(X_poly_training)\n",
    "    yhat_polynomial_lr_validation = polynomial_linear_regression.predict(X_poly_validation)\n",
    "    yhat_polynomial_lr_test       = polynomial_linear_regression.predict(X_poly_test)\n",
    "    \n",
    "    #training scores\n",
    "    r2_score_training       = mt.r2_score(y_training_regression, yhat_polynomial_lr_training)\n",
    "    mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_polynomial_lr_training)\n",
    "    rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_polynomial_lr_training))\n",
    "    mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_polynomial_lr_training)\n",
    "    mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_polynomial_lr_training)\n",
    "    \n",
    "    r2_list_training.append(r2_score_training)\n",
    "    mse_list_training.append(mse_score_training)\n",
    "    rmse_list_training.append(rmse_score_training)\n",
    "    mae_list_training.append(mae_score_training)\n",
    "    mape_list_training.append(mape_score_training)\n",
    "    \n",
    "    #validation scores\n",
    "    r2_score_validation       = mt.r2_score(y_validation_regression, yhat_polynomial_lr_validation)\n",
    "    mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_polynomial_lr_validation)\n",
    "    rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_polynomial_lr_validation))\n",
    "    mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_polynomial_lr_validation)\n",
    "    mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_polynomial_lr_validation)\n",
    "    \n",
    "    r2_list_validation.append(r2_score_validation)\n",
    "    mse_list_validation.append(mse_score_validation)\n",
    "    rmse_list_validation.append(rmse_score_validation)\n",
    "    mae_list_validation.append(mae_score_validation)\n",
    "    mape_list_validation.append(mape_score_validation)\n",
    "    \n",
    "    #test scores\n",
    "    r2_score_test       = mt.r2_score(y_test_regression, yhat_polynomial_lr_test)\n",
    "    mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_polynomial_lr_test)\n",
    "    rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_polynomial_lr_test))\n",
    "    mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_polynomial_lr_test)\n",
    "    mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_polynomial_lr_test)\n",
    "    \n",
    "    r2_list_test.append(r2_score_test)\n",
    "    mse_list_test.append(mse_score_test)\n",
    "    rmse_list_test.append(rmse_score_test)\n",
    "    mae_list_test.append(mae_score_test)\n",
    "    mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996112</td>\n",
       "      <td>21.354065</td>\n",
       "      <td>16.998249</td>\n",
       "      <td>8.653186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.094195</td>\n",
       "      <td>432.986210</td>\n",
       "      <td>20.808321</td>\n",
       "      <td>16.458032</td>\n",
       "      <td>8.350540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.154418</td>\n",
       "      <td>404.198950</td>\n",
       "      <td>20.104700</td>\n",
       "      <td>15.883592</td>\n",
       "      <td>7.800181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.333957</td>\n",
       "      <td>318.377086</td>\n",
       "      <td>17.843124</td>\n",
       "      <td>13.614247</td>\n",
       "      <td>5.913391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.725300</td>\n",
       "      <td>131.310015</td>\n",
       "      <td>11.459058</td>\n",
       "      <td>7.266166</td>\n",
       "      <td>2.215335</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degree  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0       1  0.046058  455.996112   21.354065  16.998249    8.653186\n",
       "1       2  0.094195  432.986210   20.808321  16.458032    8.350540\n",
       "2       3  0.154418  404.198950   20.104700  15.883592    7.800181\n",
       "3       4  0.333957  318.377086   17.843124  13.614247    5.913391\n",
       "4       5  0.725300  131.310015   11.459058   7.266166    2.215335"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_regression_training_df = pd.DataFrame({ 'degree': degree_estimator_list, 'r2_score': r2_list_training, \n",
    "                                                   'mse_score': mse_list_training, 'rmse_score': rmse_list_training, \n",
    "                                                   'mae_score': mae_list_training, 'mape_score': mape_list_training})\n",
    "polynomial_regression_training_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.7253</td>\n",
       "      <td>131.310015</td>\n",
       "      <td>11.459058</td>\n",
       "      <td>7.266166</td>\n",
       "      <td>2.215335</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degree  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "4       5    0.7253  131.310015   11.459058   7.266166    2.215335"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_regression_training_min = polynomial_regression_training_df.query('mape_score == mape_score.min()')\n",
    "polynomial_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.039925</td>\n",
       "      <td>4.584470e+02</td>\n",
       "      <td>21.411376</td>\n",
       "      <td>17.039754</td>\n",
       "      <td>8.682542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.066477</td>\n",
       "      <td>4.457682e+02</td>\n",
       "      <td>21.113224</td>\n",
       "      <td>16.749939</td>\n",
       "      <td>8.547931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>-0.047778</td>\n",
       "      <td>5.003263e+02</td>\n",
       "      <td>22.367974</td>\n",
       "      <td>17.087201</td>\n",
       "      <td>8.678283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>-102.923632</td>\n",
       "      <td>4.962474e+04</td>\n",
       "      <td>222.766112</td>\n",
       "      <td>36.104220</td>\n",
       "      <td>10.184802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>-224879.156426</td>\n",
       "      <td>1.073829e+08</td>\n",
       "      <td>10362.571399</td>\n",
       "      <td>1354.109130</td>\n",
       "      <td>346.300824</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degree       r2_score     mse_score    rmse_score    mae_score  mape_score\n",
       "0       1       0.039925  4.584470e+02     21.411376    17.039754    8.682542\n",
       "1       2       0.066477  4.457682e+02     21.113224    16.749939    8.547931\n",
       "2       3      -0.047778  5.003263e+02     22.367974    17.087201    8.678283\n",
       "3       4    -102.923632  4.962474e+04    222.766112    36.104220   10.184802\n",
       "4       5 -224879.156426  1.073829e+08  10362.571399  1354.109130  346.300824"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_regression_validation_df = pd.DataFrame({ 'degree': degree_estimator_list, 'r2_score': r2_list_validation, \n",
    "                                                   'mse_score': mse_list_validation, 'rmse_score': rmse_list_validation, \n",
    "                                                   'mae_score': mae_list_validation, 'mape_score': mape_list_validation})\n",
    "polynomial_regression_validation_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.066477</td>\n",
       "      <td>445.768223</td>\n",
       "      <td>21.113224</td>\n",
       "      <td>16.749939</td>\n",
       "      <td>8.547931</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degree  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "1       2  0.066477  445.768223   21.113224  16.749939    8.547931"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_regression_validation_min = polynomial_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "polynomial_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5.231712e-02</td>\n",
       "      <td>4.614277e+02</td>\n",
       "      <td>21.480869</td>\n",
       "      <td>17.129965</td>\n",
       "      <td>8.521859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>9.007934e-02</td>\n",
       "      <td>4.430413e+02</td>\n",
       "      <td>21.048545</td>\n",
       "      <td>16.720535</td>\n",
       "      <td>8.242464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>-2.617516e-01</td>\n",
       "      <td>6.143481e+02</td>\n",
       "      <td>24.786046</td>\n",
       "      <td>17.178214</td>\n",
       "      <td>7.956229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>-5.633125e+02</td>\n",
       "      <td>2.747643e+05</td>\n",
       "      <td>524.179648</td>\n",
       "      <td>40.303351</td>\n",
       "      <td>19.956479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>-1.454237e+06</td>\n",
       "      <td>7.080701e+08</td>\n",
       "      <td>26609.586203</td>\n",
       "      <td>1524.866134</td>\n",
       "      <td>465.112390</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degree      r2_score     mse_score    rmse_score    mae_score  mape_score\n",
       "0       1  5.231712e-02  4.614277e+02     21.480869    17.129965    8.521859\n",
       "1       2  9.007934e-02  4.430413e+02     21.048545    16.720535    8.242464\n",
       "2       3 -2.617516e-01  6.143481e+02     24.786046    17.178214    7.956229\n",
       "3       4 -5.633125e+02  2.747643e+05    524.179648    40.303351   19.956479\n",
       "4       5 -1.454237e+06  7.080701e+08  26609.586203  1524.866134  465.112390"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_regression_test_df = pd.DataFrame({ 'degree': degree_estimator_list, 'r2_score': r2_list_test, \n",
    "                                                   'mse_score': mse_list_test, 'rmse_score': rmse_list_test, \n",
    "                                                   'mae_score': mae_list_test, 'mape_score': mape_list_test})\n",
    "polynomial_regression_test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>-0.261752</td>\n",
       "      <td>614.348098</td>\n",
       "      <td>24.786046</td>\n",
       "      <td>17.178214</td>\n",
       "      <td>7.956229</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   degree  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "2       3 -0.261752  614.348098   24.786046  17.178214    7.956229"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_regression_test_min = polynomial_regression_test_df.query('mape_score == mape_score.min()')\n",
    "polynomial_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   The polynomial X Basis expansion is an expensive computational process as you add up more degrees to the model, hence why it was done up to degree=5 on this study.\n",
    "*   It is possible to conclude that the larger/more complex a dataset gets, higher degrees polynomial regression tends to describe the phenomenom with better accuracy, such as seen on r2 and MAPE scores for our largest dataset, the training one.\n",
    "*   When the dataset does not have enough data, its behaviour tend to converge with lower degrees, such as seen on validation and test datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.6 Polinomial Regression Lasso (L1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'degree': np.arange(1, 6).tolist(),\n",
    "          'alpha': [1, 50, 100, 200],\n",
    "          'max_iter': [1, 50, 100, 200]}\n",
    "\n",
    "degree_estimator_list = list()\n",
    "alpha_estimator_list = list()\n",
    "max_iter_estimator_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('degree'):\n",
    "    for j in params.get('alpha'):\n",
    "        for p in params.get('max_iter'):\n",
    "            degree_estimator_list.append(i)\n",
    "            alpha_estimator_list.append(j)\n",
    "            max_iter_estimator_list.append(p)\n",
    "            #Model Define\n",
    "            polynomial = pp.PolynomialFeatures(degree=i)\n",
    "            X_poly_training = polynomial.fit_transform(X_training_regression)\n",
    "            X_poly_validation = polynomial.fit_transform(X_validation_regression)\n",
    "            X_poly_test = polynomial.fit_transform(X_test_regression)\n",
    "    \n",
    "            polynomial_lasso_regression = Lasso(alpha=j, max_iter=p, random_state=42)\n",
    "    \n",
    "            #Model Training\n",
    "            polynomial_lasso_regression.fit(X_poly_training, np.ravel(y_training_regression))\n",
    "    \n",
    "            #Model Predict\n",
    "            yhat_polynomial_lasso_training   = polynomial_lasso_regression.predict(X_poly_training)\n",
    "            yhat_polynomial_lasso_validation = polynomial_lasso_regression.predict(X_poly_validation)\n",
    "            yhat_polynomial_lasso_test       = polynomial_lasso_regression.predict(X_poly_test)\n",
    "    \n",
    "            #training scores\n",
    "            r2_score_training       = mt.r2_score(y_training_regression, yhat_polynomial_lasso_training)\n",
    "            mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_polynomial_lasso_training)\n",
    "            rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_polynomial_lasso_training))\n",
    "            mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_polynomial_lasso_training)\n",
    "            mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_polynomial_lasso_training)\n",
    "    \n",
    "            r2_list_training.append(r2_score_training)\n",
    "            mse_list_training.append(mse_score_training)\n",
    "            rmse_list_training.append(rmse_score_training)\n",
    "            mae_list_training.append(mae_score_training)\n",
    "            mape_list_training.append(mape_score_training)\n",
    "            \n",
    "            #validation scores\n",
    "            r2_score_validation       = mt.r2_score(y_validation_regression, yhat_polynomial_lasso_validation)\n",
    "            mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_polynomial_lasso_validation)\n",
    "            rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_polynomial_lasso_validation))\n",
    "            mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_polynomial_lasso_validation)\n",
    "            mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_polynomial_lasso_validation)\n",
    "    \n",
    "            r2_list_validation.append(r2_score_validation)\n",
    "            mse_list_validation.append(mse_score_validation)\n",
    "            rmse_list_validation.append(rmse_score_validation)\n",
    "            mae_list_validation.append(mae_score_validation)\n",
    "            mape_list_validation.append(mape_score_validation)\n",
    "            \n",
    "            #test scores\n",
    "            r2_score_test       = mt.r2_score(y_test_regression, yhat_polynomial_lasso_test)\n",
    "            mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_polynomial_lasso_test)\n",
    "            rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_polynomial_lasso_test))\n",
    "            mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_polynomial_lasso_test)\n",
    "            mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_polynomial_lasso_test)\n",
    "    \n",
    "            r2_list_test.append(r2_score_test)\n",
    "            mse_list_test.append(mse_score_test)\n",
    "            rmse_list_test.append(rmse_score_test)\n",
    "            mae_list_test.append(mae_score_test)\n",
    "            mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>0.0195</td>\n",
       "      <td>468.691307</td>\n",
       "      <td>21.64928</td>\n",
       "      <td>17.158121</td>\n",
       "      <td>8.574792</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    degree  alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "67       5      1       200    0.0195  468.691307    21.64928  17.158121   \n",
       "\n",
       "    mape_score  \n",
       "67    8.574792  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_lasso_regression_training_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_training, \n",
    "                                                         'mse_score': mse_list_training, 'rmse_score': rmse_list_training, \n",
    "                                                         'mae_score': mae_list_training, 'mape_score': mape_list_training})\n",
    "\n",
    "polynomial_lasso_regression_training_min = polynomial_lasso_regression_training_df.query('mape_score == mape_score.min()')\n",
    "polynomial_lasso_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>0.014148</td>\n",
       "      <td>470.755769</td>\n",
       "      <td>21.696907</td>\n",
       "      <td>17.180595</td>\n",
       "      <td>8.655828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>0.014148</td>\n",
       "      <td>470.755769</td>\n",
       "      <td>21.696907</td>\n",
       "      <td>17.180595</td>\n",
       "      <td>8.655828</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    degree  alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "34       3      1       100  0.014148  470.755769   21.696907  17.180595   \n",
       "35       3      1       200  0.014148  470.755769   21.696907  17.180595   \n",
       "\n",
       "    mape_score  \n",
       "34    8.655828  \n",
       "35    8.655828  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_lasso_regression_validation_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_validation, \n",
    "                                                         'mse_score': mse_list_validation, 'rmse_score': rmse_list_validation, \n",
    "                                                         'mae_score': mae_list_validation, 'mape_score': mape_list_validation})\n",
    "\n",
    "polynomial_lasso_regression_validation_min = polynomial_lasso_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "polynomial_lasso_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>5</td>\n",
       "      <td>200</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>486.899198</td>\n",
       "      <td>22.065792</td>\n",
       "      <td>17.550363</td>\n",
       "      <td>8.714456</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    degree  alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "76       5    200         1  0.000004  486.899198   22.065792  17.550363   \n",
       "\n",
       "    mape_score  \n",
       "76    8.714456  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_lasso_regression_test_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_test, \n",
    "                                                         'mse_score': mse_list_test, 'rmse_score': rmse_list_test, \n",
    "                                                         'mae_score': mae_list_test, 'mape_score': mape_list_test})\n",
    "\n",
    "polynomial_lasso_regression_test_min = polynomial_lasso_regression_test_df.query('mape_score == mape_score.min()')\n",
    "polynomial_lasso_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.7 Polinomial Regression Ridge (L2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'degree': np.arange(1, 6).tolist(),\n",
    "          'alpha': [1, 50, 100, 200],\n",
    "          'max_iter': [1, 50, 100, 200]}\n",
    "\n",
    "degree_estimator_list = list()\n",
    "alpha_estimator_list = list()\n",
    "max_iter_estimator_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('degree'):\n",
    "    for j in params.get('alpha'):\n",
    "        for p in params.get('max_iter'):\n",
    "            degree_estimator_list.append(i)\n",
    "            alpha_estimator_list.append(j)\n",
    "            max_iter_estimator_list.append(p)\n",
    "            #Model Define\n",
    "            polynomial = pp.PolynomialFeatures(degree=i)\n",
    "            X_poly_training = polynomial.fit_transform(X_training_regression)\n",
    "            X_poly_validation = polynomial.fit_transform(X_validation_regression)\n",
    "            X_poly_test = polynomial.fit_transform(X_test_regression)\n",
    "    \n",
    "            polynomial_ridge_regression = Ridge(alpha=j, max_iter=p, random_state=42)\n",
    "    \n",
    "            #Model Training\n",
    "            polynomial_ridge_regression.fit(X_poly_training, np.ravel(y_training_regression))\n",
    "    \n",
    "            #Model Predict\n",
    "            yhat_polynomial_ridge_training   = polynomial_ridge_regression.predict(X_poly_training)\n",
    "            yhat_polynomial_ridge_validation = polynomial_ridge_regression.predict(X_poly_validation)\n",
    "            yhat_polynomial_ridge_test       = polynomial_ridge_regression.predict(X_poly_test)\n",
    "    \n",
    "            #training scores\n",
    "            r2_score_training       = mt.r2_score(y_training_regression, yhat_polynomial_ridge_training)\n",
    "            mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_polynomial_ridge_training)\n",
    "            rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_polynomial_ridge_training))\n",
    "            mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_polynomial_ridge_training)\n",
    "            mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_polynomial_ridge_training)\n",
    "    \n",
    "            r2_list_training.append(r2_score_training)\n",
    "            mse_list_training.append(mse_score_training)\n",
    "            rmse_list_training.append(rmse_score_training)\n",
    "            mae_list_training.append(mae_score_training)\n",
    "            mape_list_training.append(mape_score_training)\n",
    "            \n",
    "            #validation scores\n",
    "            r2_score_validation       = mt.r2_score(y_validation_regression, yhat_polynomial_ridge_validation)\n",
    "            mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_polynomial_ridge_validation)\n",
    "            rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_polynomial_ridge_validation))\n",
    "            mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_polynomial_ridge_validation)\n",
    "            mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_polynomial_ridge_validation)\n",
    "    \n",
    "            r2_list_validation.append(r2_score_validation)\n",
    "            mse_list_validation.append(mse_score_validation)\n",
    "            rmse_list_validation.append(rmse_score_validation)\n",
    "            mae_list_validation.append(mae_score_validation)\n",
    "            mape_list_validation.append(mape_score_validation)\n",
    "            \n",
    "            #test scores\n",
    "            r2_score_test       = mt.r2_score(y_test_regression, yhat_polynomial_ridge_test)\n",
    "            mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_polynomial_ridge_test)\n",
    "            rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_polynomial_ridge_test))\n",
    "            mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_polynomial_ridge_test)\n",
    "            mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_polynomial_ridge_test)\n",
    "    \n",
    "            r2_list_test.append(r2_score_test)\n",
    "            mse_list_test.append(mse_score_test)\n",
    "            rmse_list_test.append(rmse_score_test)\n",
    "            mae_list_test.append(mae_score_test)\n",
    "            mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.312725</td>\n",
       "      <td>328.525923</td>\n",
       "      <td>18.125284</td>\n",
       "      <td>14.002097</td>\n",
       "      <td>6.477741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>0.312725</td>\n",
       "      <td>328.525923</td>\n",
       "      <td>18.125284</td>\n",
       "      <td>14.002097</td>\n",
       "      <td>6.477741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>0.312725</td>\n",
       "      <td>328.525923</td>\n",
       "      <td>18.125284</td>\n",
       "      <td>14.002097</td>\n",
       "      <td>6.477741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>0.312725</td>\n",
       "      <td>328.525923</td>\n",
       "      <td>18.125284</td>\n",
       "      <td>14.002097</td>\n",
       "      <td>6.477741</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    degree  alpha  max_iter  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "64       5      1         1  0.312725  328.525923   18.125284  14.002097   \n",
       "65       5      1        50  0.312725  328.525923   18.125284  14.002097   \n",
       "66       5      1       100  0.312725  328.525923   18.125284  14.002097   \n",
       "67       5      1       200  0.312725  328.525923   18.125284  14.002097   \n",
       "\n",
       "    mape_score  \n",
       "64    6.477741  \n",
       "65    6.477741  \n",
       "66    6.477741  \n",
       "67    6.477741  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_ridge_regression_training_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_training, \n",
    "                                                         'mse_score': mse_list_training, 'rmse_score': rmse_list_training, \n",
    "                                                         'mae_score': mae_list_training, 'mape_score': mape_list_training})\n",
    "\n",
    "polynomial_ridge_regression_training_min = polynomial_ridge_regression_training_df.query('mape_score == mape_score.min()')\n",
    "polynomial_ridge_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>5</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>-9.685894</td>\n",
       "      <td>5102.638358</td>\n",
       "      <td>71.432754</td>\n",
       "      <td>18.826258</td>\n",
       "      <td>8.348152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>5</td>\n",
       "      <td>50</td>\n",
       "      <td>50</td>\n",
       "      <td>-9.685894</td>\n",
       "      <td>5102.638358</td>\n",
       "      <td>71.432754</td>\n",
       "      <td>18.826258</td>\n",
       "      <td>8.348152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>5</td>\n",
       "      <td>50</td>\n",
       "      <td>100</td>\n",
       "      <td>-9.685894</td>\n",
       "      <td>5102.638358</td>\n",
       "      <td>71.432754</td>\n",
       "      <td>18.826258</td>\n",
       "      <td>8.348152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>5</td>\n",
       "      <td>50</td>\n",
       "      <td>200</td>\n",
       "      <td>-9.685894</td>\n",
       "      <td>5102.638358</td>\n",
       "      <td>71.432754</td>\n",
       "      <td>18.826258</td>\n",
       "      <td>8.348152</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    degree  alpha  max_iter  r2_score    mse_score  rmse_score  mae_score  \\\n",
       "68       5     50         1 -9.685894  5102.638358   71.432754  18.826258   \n",
       "69       5     50        50 -9.685894  5102.638358   71.432754  18.826258   \n",
       "70       5     50       100 -9.685894  5102.638358   71.432754  18.826258   \n",
       "71       5     50       200 -9.685894  5102.638358   71.432754  18.826258   \n",
       "\n",
       "    mape_score  \n",
       "68    8.348152  \n",
       "69    8.348152  \n",
       "70    8.348152  \n",
       "71    8.348152  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_ridge_regression_validation_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_validation, \n",
    "                                                         'mse_score': mse_list_validation, 'rmse_score': rmse_list_validation, \n",
    "                                                         'mae_score': mae_list_validation, 'mape_score': mape_list_validation})\n",
    "\n",
    "polynomial_ridge_regression_validation_min = polynomial_ridge_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "polynomial_ridge_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-159.704238</td>\n",
       "      <td>78247.050076</td>\n",
       "      <td>279.726742</td>\n",
       "      <td>21.433575</td>\n",
       "      <td>8.113779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>-159.704238</td>\n",
       "      <td>78247.050076</td>\n",
       "      <td>279.726742</td>\n",
       "      <td>21.433575</td>\n",
       "      <td>8.113779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>100</td>\n",
       "      <td>-159.704238</td>\n",
       "      <td>78247.050076</td>\n",
       "      <td>279.726742</td>\n",
       "      <td>21.433575</td>\n",
       "      <td>8.113779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>-159.704238</td>\n",
       "      <td>78247.050076</td>\n",
       "      <td>279.726742</td>\n",
       "      <td>21.433575</td>\n",
       "      <td>8.113779</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    degree  alpha  max_iter    r2_score     mse_score  rmse_score  mae_score  \\\n",
       "48       4      1         1 -159.704238  78247.050076  279.726742  21.433575   \n",
       "49       4      1        50 -159.704238  78247.050076  279.726742  21.433575   \n",
       "50       4      1       100 -159.704238  78247.050076  279.726742  21.433575   \n",
       "51       4      1       200 -159.704238  78247.050076  279.726742  21.433575   \n",
       "\n",
       "    mape_score  \n",
       "48    8.113779  \n",
       "49    8.113779  \n",
       "50    8.113779  \n",
       "51    8.113779  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_ridge_regression_test_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                         'max_iter': max_iter_estimator_list, 'r2_score': r2_list_test, \n",
    "                                                         'mse_score': mse_list_test, 'rmse_score': rmse_list_test, \n",
    "                                                         'mae_score': mae_list_test, 'mape_score': mape_list_test})\n",
    "\n",
    "polynomial_ridge_regression_test_min = polynomial_ridge_regression_test_df.query('mape_score == mape_score.min()')\n",
    "polynomial_ridge_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Both Lasso and Ridge Regression , like in simple Linear Regression model, tend to raise up the bias on the polynomial regression. This can be seen by how regularized polynomial regression values for r2 and MAPE scores varies less in comparison to simple polynomial regression.\n",
    "*   By having less variance error, we can then see that Polynomial Regularized Regression models are able to better generalize data behaviour."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.8 Polinomial Regression Elastic Net (L1 & L2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'degree': np.arange(1, 6).tolist(),\n",
    "          'alpha': [1, 50, 100, 200],\n",
    "          'l1_ratio': [0, 0.2, 0.5, 0.7, 1.0],\n",
    "          'max_iter': [1, 50, 100, 200]}\n",
    "\n",
    "degree_estimator_list = list()\n",
    "alpha_estimator_list = list()\n",
    "l1_ratio_estimator_list = list()\n",
    "max_iter_estimator_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('degree'):\n",
    "    for j in params.get('alpha'):\n",
    "        for p in params.get('l1_ratio'):\n",
    "            for q in params.get('max_iter'):\n",
    "                degree_estimator_list.append(i)\n",
    "                alpha_estimator_list.append(j)\n",
    "                l1_ratio_estimator_list.append(p)\n",
    "                max_iter_estimator_list.append(q)\n",
    "                #Model Define\n",
    "                polynomial = pp.PolynomialFeatures(degree=i)\n",
    "                X_poly_training = polynomial.fit_transform(X_training_regression)\n",
    "                X_poly_validation = polynomial.fit_transform(X_validation_regression)\n",
    "                X_poly_test = polynomial.fit_transform(X_test_regression)\n",
    "    \n",
    "                polynomial_elastic_net_regression = ElasticNet(alpha=j, l1_ratio=p, max_iter=q, random_state=42)\n",
    "    \n",
    "                #Model Training\n",
    "                polynomial_elastic_net_regression.fit(X_poly_training, np.ravel(y_training_regression))\n",
    "    \n",
    "                #Model Predict\n",
    "                yhat_polynomial_elastic_net_training   = polynomial_elastic_net_regression.predict(X_poly_training)\n",
    "                yhat_polynomial_elastic_net_validation = polynomial_elastic_net_regression.predict(X_poly_validation)\n",
    "                yhat_polynomial_elastic_net_test       = polynomial_elastic_net_regression.predict(X_poly_test)\n",
    "                \n",
    "                #training scores\n",
    "                r2_score_training       = mt.r2_score(y_training_regression, yhat_polynomial_elastic_net_training)\n",
    "                mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_polynomial_elastic_net_training)\n",
    "                rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_polynomial_elastic_net_training))\n",
    "                mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_polynomial_elastic_net_training)\n",
    "                mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_polynomial_elastic_net_training)\n",
    "    \n",
    "                r2_list_training.append(r2_score_training)\n",
    "                mse_list_training.append(mse_score_training)\n",
    "                rmse_list_training.append(rmse_score_training)\n",
    "                mae_list_training.append(mae_score_training)\n",
    "                mape_list_training.append(mape_score_training)\n",
    "                \n",
    "                #validation scores\n",
    "                r2_score_validation       = mt.r2_score(y_validation_regression, yhat_polynomial_elastic_net_validation)\n",
    "                mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_polynomial_elastic_net_validation)\n",
    "                rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_polynomial_elastic_net_validation))\n",
    "                mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_polynomial_elastic_net_validation)\n",
    "                mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_polynomial_elastic_net_validation)\n",
    "    \n",
    "                r2_list_validation.append(r2_score_validation)\n",
    "                mse_list_validation.append(mse_score_validation)\n",
    "                rmse_list_validation.append(rmse_score_validation)\n",
    "                mae_list_validation.append(mae_score_validation)\n",
    "                mape_list_validation.append(mape_score_validation)  \n",
    "                \n",
    "                #test scores\n",
    "                r2_score_test       = mt.r2_score(y_test_regression, yhat_polynomial_elastic_net_test)\n",
    "                mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_polynomial_elastic_net_test)\n",
    "                rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_polynomial_elastic_net_test))\n",
    "                mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_polynomial_elastic_net_test)\n",
    "                mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_polynomial_elastic_net_test)\n",
    "    \n",
    "                r2_list_test.append(r2_score_test)\n",
    "                mse_list_test.append(mse_score_test)\n",
    "                rmse_list_test.append(rmse_score_test)\n",
    "                mae_list_test.append(mae_score_test)\n",
    "                mape_list_test.append(mape_score_test)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>l1_ratio</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>323</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>200</td>\n",
       "      <td>0.085699</td>\n",
       "      <td>437.047167</td>\n",
       "      <td>20.905673</td>\n",
       "      <td>16.523671</td>\n",
       "      <td>8.278886</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     degree  alpha  l1_ratio  max_iter  r2_score   mse_score  rmse_score  \\\n",
       "323       5      1       0.0       200  0.085699  437.047167   20.905673   \n",
       "\n",
       "     mae_score  mape_score  \n",
       "323  16.523671    8.278886  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_elastic_net_regression_training_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                               'l1_ratio': l1_ratio_estimator_list, 'max_iter': max_iter_estimator_list, \n",
    "                                                               'r2_score': r2_list_training, 'mse_score': mse_list_training, \n",
    "                                                               'rmse_score': rmse_list_training, 'mae_score': mae_list_training, \n",
    "                                                               'mape_score': mape_list_training})\n",
    "\n",
    "polynomial_elastic_net_regression_training_min = polynomial_elastic_net_regression_training_df.query('mape_score == mape_score.min()')\n",
    "polynomial_elastic_net_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>l1_ratio</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.054574</td>\n",
       "      <td>451.452047</td>\n",
       "      <td>21.247401</td>\n",
       "      <td>16.808199</td>\n",
       "      <td>8.630816</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     degree  alpha  l1_ratio  max_iter  r2_score   mse_score  rmse_score  \\\n",
       "320       5      1       0.0         1  0.054574  451.452047   21.247401   \n",
       "\n",
       "     mae_score  mape_score  \n",
       "320  16.808199    8.630816  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_elastic_net_regression_validation_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                                 'l1_ratio': l1_ratio_estimator_list, 'max_iter': max_iter_estimator_list, \n",
    "                                                                 'r2_score': r2_list_validation, 'mse_score': mse_list_validation, \n",
    "                                                                 'rmse_score': rmse_list_validation, 'mae_score': mae_list_validation, \n",
    "                                                                 'mape_score': mape_list_validation})\n",
    "\n",
    "polynomial_elastic_net_regression_validation_min = polynomial_elastic_net_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "polynomial_elastic_net_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>degree</th>\n",
       "      <th>alpha</th>\n",
       "      <th>l1_ratio</th>\n",
       "      <th>max_iter</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.169653</td>\n",
       "      <td>569.505007</td>\n",
       "      <td>23.864304</td>\n",
       "      <td>16.998383</td>\n",
       "      <td>8.488322</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     degree  alpha  l1_ratio  max_iter  r2_score   mse_score  rmse_score  \\\n",
       "320       5      1       0.0         1 -0.169653  569.505007   23.864304   \n",
       "\n",
       "     mae_score  mape_score  \n",
       "320  16.998383    8.488322  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polynomial_elastic_net_regression_test_df = pd.DataFrame({ 'degree': degree_estimator_list, 'alpha': alpha_estimator_list,\n",
    "                                                           'l1_ratio': l1_ratio_estimator_list, 'max_iter': max_iter_estimator_list, \n",
    "                                                           'r2_score': r2_list_test, 'mse_score': mse_list_test, \n",
    "                                                           'rmse_score': rmse_list_test, 'mae_score': mae_list_test, \n",
    "                                                           'mape_score': mape_list_test})\n",
    "\n",
    "polynomial_elastic_net_regression_test_min = polynomial_elastic_net_regression_test_df.query('mape_score == mape_score.min()')\n",
    "polynomial_elastic_net_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.9 Decision Tree Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'max_depth': [1, 5, 10, 15, 20]}\n",
    "\n",
    "max_depth_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('max_depth'):\n",
    "    max_depth_list.append(i)\n",
    "    #Model Define\n",
    "    tree_regression = DecisionTreeRegressor(max_depth=i, random_state=42)\n",
    "    \n",
    "    #Model Training\n",
    "    tree_regression.fit(X_training_regression, np.ravel(y_training_regression))\n",
    "    \n",
    "    #Model Predict\n",
    "    yhat_tree_regression_training   = tree_regression.predict(X_training_regression)\n",
    "    yhat_tree_regression_validation = tree_regression.predict(X_validation_regression)\n",
    "    yhat_tree_regression_test       = tree_regression.predict(X_test_regression)\n",
    "    \n",
    "    #training scores\n",
    "    r2_score_training       = mt.r2_score(y_training_regression, yhat_tree_regression_training)\n",
    "    mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_tree_regression_training)\n",
    "    rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_tree_regression_training))\n",
    "    mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_tree_regression_training)\n",
    "    mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_tree_regression_training)\n",
    "    \n",
    "    r2_list_training.append(r2_score_training)\n",
    "    mse_list_training.append(mse_score_training)\n",
    "    rmse_list_training.append(rmse_score_training)\n",
    "    mae_list_training.append(mae_score_training)\n",
    "    mape_list_training.append(mape_score_training)\n",
    "    \n",
    "    #validation scores\n",
    "    r2_score_validation       = mt.r2_score(y_validation_regression, yhat_tree_regression_validation)\n",
    "    mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_tree_regression_validation)\n",
    "    rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_tree_regression_validation))\n",
    "    mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_tree_regression_validation)\n",
    "    mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_tree_regression_validation)\n",
    "    \n",
    "    r2_list_validation.append(r2_score_validation)\n",
    "    mse_list_validation.append(mse_score_validation)\n",
    "    rmse_list_validation.append(rmse_score_validation)\n",
    "    mae_list_validation.append(mae_score_validation)\n",
    "    mape_list_validation.append(mape_score_validation)\n",
    "    \n",
    "    #test scores\n",
    "    r2_score_test       = mt.r2_score(y_test_regression, yhat_tree_regression_test)\n",
    "    mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_tree_regression_test)\n",
    "    rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_tree_regression_test))\n",
    "    mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_tree_regression_test)\n",
    "    mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_tree_regression_test)\n",
    "    \n",
    "    r2_list_test.append(r2_score_test)\n",
    "    mse_list_test.append(mse_score_test)\n",
    "    rmse_list_test.append(rmse_score_test)\n",
    "    mae_list_test.append(mae_score_test)\n",
    "    mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.025572</td>\n",
       "      <td>465.788730</td>\n",
       "      <td>21.582139</td>\n",
       "      <td>17.158819</td>\n",
       "      <td>8.615678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>0.113523</td>\n",
       "      <td>423.747268</td>\n",
       "      <td>20.585122</td>\n",
       "      <td>16.368766</td>\n",
       "      <td>7.869536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>0.384624</td>\n",
       "      <td>294.157341</td>\n",
       "      <td>17.151016</td>\n",
       "      <td>12.925051</td>\n",
       "      <td>4.871411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>0.738302</td>\n",
       "      <td>125.095124</td>\n",
       "      <td>11.184593</td>\n",
       "      <td>6.603326</td>\n",
       "      <td>2.049551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>0.927211</td>\n",
       "      <td>34.794149</td>\n",
       "      <td>5.898657</td>\n",
       "      <td>2.144293</td>\n",
       "      <td>0.356446</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0          1  0.025572  465.788730   21.582139  17.158819    8.615678\n",
       "1          5  0.113523  423.747268   20.585122  16.368766    7.869536\n",
       "2         10  0.384624  294.157341   17.151016  12.925051    4.871411\n",
       "3         15  0.738302  125.095124   11.184593   6.603326    2.049551\n",
       "4         20  0.927211   34.794149    5.898657   2.144293    0.356446"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_regression_training_df = pd.DataFrame({ 'max_depth': max_depth_list, 'r2_score': r2_list_training, \n",
    "                                             'mse_score': mse_list_training, 'rmse_score': rmse_list_training, \n",
    "                                             'mae_score': mae_list_training, 'mape_score': mape_list_training})\n",
    "tree_regression_training_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>0.927211</td>\n",
       "      <td>34.794149</td>\n",
       "      <td>5.898657</td>\n",
       "      <td>2.144293</td>\n",
       "      <td>0.356446</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  r2_score  mse_score  rmse_score  mae_score  mape_score\n",
       "4         20  0.927211  34.794149    5.898657   2.144293    0.356446"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_regression_training_min = tree_regression_training_df.query('mape_score == mape_score.min()')\n",
    "tree_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.025733</td>\n",
       "      <td>465.223863</td>\n",
       "      <td>21.569049</td>\n",
       "      <td>17.122340</td>\n",
       "      <td>8.549934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>0.063559</td>\n",
       "      <td>447.161319</td>\n",
       "      <td>21.146189</td>\n",
       "      <td>16.843452</td>\n",
       "      <td>8.395778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>-0.002935</td>\n",
       "      <td>478.912947</td>\n",
       "      <td>21.884080</td>\n",
       "      <td>16.861326</td>\n",
       "      <td>7.879333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>-0.133504</td>\n",
       "      <td>541.261389</td>\n",
       "      <td>23.265025</td>\n",
       "      <td>16.880678</td>\n",
       "      <td>7.678007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>-0.251930</td>\n",
       "      <td>597.811099</td>\n",
       "      <td>24.450176</td>\n",
       "      <td>17.132240</td>\n",
       "      <td>7.054840</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0          1  0.025733  465.223863   21.569049  17.122340    8.549934\n",
       "1          5  0.063559  447.161319   21.146189  16.843452    8.395778\n",
       "2         10 -0.002935  478.912947   21.884080  16.861326    7.879333\n",
       "3         15 -0.133504  541.261389   23.265025  16.880678    7.678007\n",
       "4         20 -0.251930  597.811099   24.450176  17.132240    7.054840"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_regression_validation_df = pd.DataFrame({ 'max_depth': max_depth_list, 'r2_score': r2_list_validation, \n",
    "                                             'mse_score': mse_list_validation, 'rmse_score': rmse_list_validation, \n",
    "                                             'mae_score': mae_list_validation, 'mape_score': mape_list_validation})\n",
    "tree_regression_validation_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>-0.25193</td>\n",
       "      <td>597.811099</td>\n",
       "      <td>24.450176</td>\n",
       "      <td>17.13224</td>\n",
       "      <td>7.05484</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "4         20  -0.25193  597.811099   24.450176   17.13224     7.05484"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_regression_validation_min = tree_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "tree_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.024996</td>\n",
       "      <td>474.730587</td>\n",
       "      <td>21.788313</td>\n",
       "      <td>17.381588</td>\n",
       "      <td>8.519214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>0.072181</td>\n",
       "      <td>451.755789</td>\n",
       "      <td>21.254547</td>\n",
       "      <td>17.010757</td>\n",
       "      <td>7.833952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>0.045693</td>\n",
       "      <td>464.652786</td>\n",
       "      <td>21.555806</td>\n",
       "      <td>16.743015</td>\n",
       "      <td>7.062497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>-0.078512</td>\n",
       "      <td>525.128765</td>\n",
       "      <td>22.915688</td>\n",
       "      <td>16.871659</td>\n",
       "      <td>6.460198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>-0.192656</td>\n",
       "      <td>580.705338</td>\n",
       "      <td>24.097828</td>\n",
       "      <td>16.961002</td>\n",
       "      <td>6.358549</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "0          1  0.024996  474.730587   21.788313  17.381588    8.519214\n",
       "1          5  0.072181  451.755789   21.254547  17.010757    7.833952\n",
       "2         10  0.045693  464.652786   21.555806  16.743015    7.062497\n",
       "3         15 -0.078512  525.128765   22.915688  16.871659    6.460198\n",
       "4         20 -0.192656  580.705338   24.097828  16.961002    6.358549"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_regression_test_df = pd.DataFrame({ 'max_depth': max_depth_list, 'r2_score': r2_list_test, \n",
    "                                             'mse_score': mse_list_test, 'rmse_score': rmse_list_test, \n",
    "                                             'mae_score': mae_list_test, 'mape_score': mape_list_test})\n",
    "tree_regression_test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>-0.192656</td>\n",
       "      <td>580.705338</td>\n",
       "      <td>24.097828</td>\n",
       "      <td>16.961002</td>\n",
       "      <td>6.358549</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  r2_score   mse_score  rmse_score  mae_score  mape_score\n",
       "4         20 -0.192656  580.705338   24.097828  16.961002    6.358549"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_regression_test_min = tree_regression_test_df.query('mape_score == mape_score.min()')\n",
    "tree_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   As with Decision Tree Classifier, the more depth you add to a tree model, the more it tends to overfit, hence the difference in results between train and validation/test datasets for MAPE score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.10 Random Forest Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'n_estimators':[1, 10, 50, 100, 150, 200],\n",
    "          'max_depth': [1, 5, 10, 15, 20]}\n",
    "\n",
    "n_estimators_list = list()\n",
    "max_depth_list = list()\n",
    "\n",
    "r2_list_training = list()\n",
    "mse_list_training = list()\n",
    "rmse_list_training = list()\n",
    "mae_list_training = list()\n",
    "mape_list_training = list()\n",
    "\n",
    "r2_list_validation = list()\n",
    "mse_list_validation = list()\n",
    "rmse_list_validation = list()\n",
    "mae_list_validation = list()\n",
    "mape_list_validation = list()\n",
    "\n",
    "r2_list_test = list()\n",
    "mse_list_test = list()\n",
    "rmse_list_test = list()\n",
    "mae_list_test = list()\n",
    "mape_list_test = list()\n",
    "\n",
    "for i in params.get('n_estimators'):\n",
    "    for j in params.get('max_depth'):\n",
    "        n_estimators_list.append(i)\n",
    "        max_depth_list.append(j)\n",
    "        #Model Define\n",
    "        rf_regression = RandomForestRegressor(n_estimators=i ,max_depth=j, random_state=42)\n",
    "    \n",
    "        #Model Training\n",
    "        rf_regression.fit(X_training_regression, np.ravel(y_training_regression))\n",
    "    \n",
    "        #Model Predict\n",
    "        yhat_rf_regression_training   = rf_regression.predict(X_training_regression)\n",
    "        yhat_rf_regression_validation = rf_regression.predict(X_validation_regression)\n",
    "        yhat_rf_regression_test       = rf_regression.predict(X_test_regression)\n",
    "    \n",
    "        #training scores\n",
    "        r2_score_training       = mt.r2_score(y_training_regression, yhat_rf_regression_training)\n",
    "        mse_score_training      = mt.mean_squared_error(y_training_regression, yhat_rf_regression_training)\n",
    "        rmse_score_training     = np.sqrt(mt.mean_squared_error(y_training_regression, yhat_rf_regression_training))\n",
    "        mae_score_training      = mt.mean_absolute_error(y_training_regression, yhat_rf_regression_training)\n",
    "        mape_score_training     = mt.mean_absolute_percentage_error(y_training_regression, yhat_rf_regression_training)\n",
    "    \n",
    "        r2_list_training.append(r2_score_training)\n",
    "        mse_list_training.append(mse_score_training)\n",
    "        rmse_list_training.append(rmse_score_training)\n",
    "        mae_list_training.append(mae_score_training)\n",
    "        mape_list_training.append(mape_score_training)\n",
    "        \n",
    "        #validation scores\n",
    "        r2_score_validation       = mt.r2_score(y_validation_regression, yhat_rf_regression_validation)\n",
    "        mse_score_validation      = mt.mean_squared_error(y_validation_regression, yhat_rf_regression_validation)\n",
    "        rmse_score_validation     = np.sqrt(mt.mean_squared_error(y_validation_regression, yhat_rf_regression_validation))\n",
    "        mae_score_validation      = mt.mean_absolute_error(y_validation_regression, yhat_rf_regression_validation)\n",
    "        mape_score_validation     = mt.mean_absolute_percentage_error(y_validation_regression, yhat_rf_regression_validation)\n",
    "    \n",
    "        r2_list_validation.append(r2_score_validation)\n",
    "        mse_list_validation.append(mse_score_validation)\n",
    "        rmse_list_validation.append(rmse_score_validation)\n",
    "        mae_list_validation.append(mae_score_validation)\n",
    "        mape_list_validation.append(mape_score_validation)\n",
    "        \n",
    "        #test scores\n",
    "        r2_score_test       = mt.r2_score(y_test_regression, yhat_rf_regression_test)\n",
    "        mse_score_test      = mt.mean_squared_error(y_test_regression, yhat_rf_regression_test)\n",
    "        rmse_score_test     = np.sqrt(mt.mean_squared_error(y_test_regression, yhat_rf_regression_test))\n",
    "        mae_score_test      = mt.mean_absolute_error(y_test_regression, yhat_rf_regression_test)\n",
    "        mape_score_test     = mt.mean_absolute_percentage_error(y_test_regression, yhat_rf_regression_test)\n",
    "    \n",
    "        r2_list_test.append(r2_score_test)\n",
    "        mse_list_test.append(mse_score_test)\n",
    "        rmse_list_test.append(rmse_score_test)\n",
    "        mae_list_test.append(mae_score_test)\n",
    "        mape_list_test.append(mape_score_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>100</td>\n",
       "      <td>20</td>\n",
       "      <td>0.883119</td>\n",
       "      <td>55.87046</td>\n",
       "      <td>7.474654</td>\n",
       "      <td>5.517068</td>\n",
       "      <td>2.682385</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    n_estimators  max_depth  r2_score  mse_score  rmse_score  mae_score  \\\n",
       "19           100         20  0.883119   55.87046    7.474654   5.517068   \n",
       "\n",
       "    mape_score  \n",
       "19    2.682385  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_regression_training_df = pd.DataFrame({ 'n_estimators': n_estimators_list, 'max_depth': max_depth_list, \n",
    "                                           'r2_score': r2_list_training, 'mse_score': mse_list_training, \n",
    "                                           'rmse_score': rmse_list_training, 'mae_score': mae_list_training, \n",
    "                                           'mape_score': mape_list_training})\n",
    "\n",
    "rf_regression_training_min = rf_regression_training_df.query('mape_score == mape_score.min()')\n",
    "rf_regression_training_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>50</td>\n",
       "      <td>20</td>\n",
       "      <td>0.321033</td>\n",
       "      <td>324.214408</td>\n",
       "      <td>18.005955</td>\n",
       "      <td>13.29633</td>\n",
       "      <td>7.066447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    n_estimators  max_depth  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "14            50         20  0.321033  324.214408   18.005955   13.29633   \n",
       "\n",
       "    mape_score  \n",
       "14    7.066447  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_regression_validation_df = pd.DataFrame({ 'n_estimators': n_estimators_list, 'max_depth': max_depth_list, \n",
    "                                             'r2_score': r2_list_validation, 'mse_score': mse_list_validation, \n",
    "                                             'rmse_score': rmse_list_validation, 'mae_score': mae_list_validation, \n",
    "                                             'mape_score': mape_list_validation})\n",
    "\n",
    "rf_regression_validation_min = rf_regression_validation_df.query('mape_score == mape_score.min()')\n",
    "rf_regression_validation_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>20</td>\n",
       "      <td>0.28078</td>\n",
       "      <td>350.188801</td>\n",
       "      <td>18.713332</td>\n",
       "      <td>13.855007</td>\n",
       "      <td>6.514113</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   n_estimators  max_depth  r2_score   mse_score  rmse_score  mae_score  \\\n",
       "9            10         20   0.28078  350.188801   18.713332  13.855007   \n",
       "\n",
       "   mape_score  \n",
       "9    6.514113  "
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_regression_test_df = pd.DataFrame({ 'n_estimators': n_estimators_list, 'max_depth': max_depth_list, \n",
    "                                       'r2_score': r2_list_test, 'mse_score': mse_list_test, \n",
    "                                       'rmse_score': rmse_list_test, 'mae_score': mae_list_test, \n",
    "                                       'mape_score': mape_list_test})\n",
    "\n",
    "rf_regression_test_min = rf_regression_test_df.query('mape_score == mape_score.min()')\n",
    "rf_regression_test_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   The Random Forest, alike the Singular Decision Tree Model, tend to overfit with the increase of estimators and tree depth.\n",
    "*   In comparison to the singular decision tree model, the Random Forest shows less variation between results from different datasets (better generalization)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.11 Final Results for Regression Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.11.1 Training Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression_training_min['model'] = 'Linear Regression'\n",
    "linear_regression_training_min = linear_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "lasso_regression_training_min = lasso_regression_training_min.drop(['alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "lasso_regression_training_min['model'] = 'Linear Regression Lasso (L1)'\n",
    "lasso_regression_training_min = lasso_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "ridge_regression_training_min = ridge_regression_training_min.drop(['alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "ridge_regression_training_min['model'] = 'Linear Regression Ridge (L2)'\n",
    "ridge_regression_training_min = ridge_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "elastic_net_training_min = elastic_net_training_min.drop(['alpha', 'l1_ratio','max_iter'], axis=1).iloc[[0]]\n",
    "elastic_net_training_min['model'] = 'Linear Regression Elastic Net (L1 & L2)'\n",
    "elastic_net_training_min = elastic_net_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_regression_training_min = polynomial_regression_training_min.drop('degree', axis=1).iloc[[0]]\n",
    "polynomial_regression_training_min['model'] = 'Polynomial Regression'\n",
    "polynomial_regression_training_min = polynomial_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_lasso_regression_training_min = polynomial_lasso_regression_training_min.drop(['degree','alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_lasso_regression_training_min['model'] = 'Polynomial Regression Lasso (L1)'\n",
    "polynomial_lasso_regression_training_min = polynomial_lasso_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_ridge_regression_training_min = polynomial_ridge_regression_training_min.drop(['degree','alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_ridge_regression_training_min['model'] = 'Polynomial Regression Ridge (L2)'\n",
    "polynomial_ridge_regression_training_min = polynomial_ridge_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_elastic_net_regression_training_min = polynomial_elastic_net_regression_training_min.drop(['degree', 'alpha', 'l1_ratio', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_elastic_net_regression_training_min['model'] = 'Polynomial Regression Elastic Net (L1 & L2)'\n",
    "polynomial_elastic_net_regression_training_min = polynomial_elastic_net_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "tree_regression_training_min = tree_regression_training_min.drop('max_depth', axis=1).iloc[[0]]\n",
    "tree_regression_training_min['model'] = 'Decision Tree Regression'\n",
    "tree_regression_training_min = tree_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "rf_regression_training_min = rf_regression_training_min.drop(['n_estimators', 'max_depth'], axis=1).iloc[[0]]\n",
    "rf_regression_training_min['model'] = 'Random Forest Regression'\n",
    "rf_regression_training_min = rf_regression_training_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Linear Regression</td>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996112</td>\n",
       "      <td>21.354065</td>\n",
       "      <td>16.998249</td>\n",
       "      <td>8.653186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Linear Regression Lasso (L1)</td>\n",
       "      <td>0.007401</td>\n",
       "      <td>474.474834</td>\n",
       "      <td>21.782443</td>\n",
       "      <td>17.305484</td>\n",
       "      <td>8.736697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Linear Regression Ridge (L2)</td>\n",
       "      <td>0.046058</td>\n",
       "      <td>455.996401</td>\n",
       "      <td>21.354072</td>\n",
       "      <td>16.998308</td>\n",
       "      <td>8.653415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Linear Regression Elastic Net (L1 &amp; L2)</td>\n",
       "      <td>0.010715</td>\n",
       "      <td>472.890766</td>\n",
       "      <td>21.746052</td>\n",
       "      <td>17.277098</td>\n",
       "      <td>8.719880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Polynomial Regression</td>\n",
       "      <td>0.725300</td>\n",
       "      <td>131.310015</td>\n",
       "      <td>11.459058</td>\n",
       "      <td>7.266166</td>\n",
       "      <td>2.215335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Polynomial Regression Lasso (L1)</td>\n",
       "      <td>0.019500</td>\n",
       "      <td>468.691307</td>\n",
       "      <td>21.649280</td>\n",
       "      <td>17.158121</td>\n",
       "      <td>8.574792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Polynomial Regression Ridge (L2)</td>\n",
       "      <td>0.312725</td>\n",
       "      <td>328.525923</td>\n",
       "      <td>18.125284</td>\n",
       "      <td>14.002097</td>\n",
       "      <td>6.477741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Polynomial Regression Elastic Net (L1 &amp; L2)</td>\n",
       "      <td>0.085699</td>\n",
       "      <td>437.047167</td>\n",
       "      <td>20.905673</td>\n",
       "      <td>16.523671</td>\n",
       "      <td>8.278886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Decision Tree Regression</td>\n",
       "      <td>0.927211</td>\n",
       "      <td>34.794149</td>\n",
       "      <td>5.898657</td>\n",
       "      <td>2.144293</td>\n",
       "      <td>0.356446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Random Forest Regression</td>\n",
       "      <td>0.883119</td>\n",
       "      <td>55.870460</td>\n",
       "      <td>7.474654</td>\n",
       "      <td>5.517068</td>\n",
       "      <td>2.682385</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         model  r2_score   mse_score  \\\n",
       "0                            Linear Regression  0.046058  455.996112   \n",
       "1                 Linear Regression Lasso (L1)  0.007401  474.474834   \n",
       "2                 Linear Regression Ridge (L2)  0.046058  455.996401   \n",
       "3      Linear Regression Elastic Net (L1 & L2)  0.010715  472.890766   \n",
       "4                        Polynomial Regression  0.725300  131.310015   \n",
       "5             Polynomial Regression Lasso (L1)  0.019500  468.691307   \n",
       "6             Polynomial Regression Ridge (L2)  0.312725  328.525923   \n",
       "7  Polynomial Regression Elastic Net (L1 & L2)  0.085699  437.047167   \n",
       "8                     Decision Tree Regression  0.927211   34.794149   \n",
       "9                     Random Forest Regression  0.883119   55.870460   \n",
       "\n",
       "   rmse_score  mae_score  mape_score  \n",
       "0   21.354065  16.998249    8.653186  \n",
       "1   21.782443  17.305484    8.736697  \n",
       "2   21.354072  16.998308    8.653415  \n",
       "3   21.746052  17.277098    8.719880  \n",
       "4   11.459058   7.266166    2.215335  \n",
       "5   21.649280  17.158121    8.574792  \n",
       "6   18.125284  14.002097    6.477741  \n",
       "7   20.905673  16.523671    8.278886  \n",
       "8    5.898657   2.144293    0.356446  \n",
       "9    7.474654   5.517068    2.682385  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_regression_results = pd.concat([linear_regression_training_min, lasso_regression_training_min, ridge_regression_training_min,\n",
    "                                         elastic_net_training_min, polynomial_regression_training_min, polynomial_lasso_regression_training_min,\n",
    "                                         polynomial_ridge_regression_training_min, polynomial_elastic_net_regression_training_min,\n",
    "                                         tree_regression_training_min, rf_regression_training_min])\n",
    "training_regression_results = training_regression_results.reset_index(drop=True)\n",
    "training_regression_results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.11.2 Validation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression_validation_min['model'] = 'Linear Regression'\n",
    "linear_regression_validation_min = linear_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "lasso_regression_validation_min = lasso_regression_validation_min.drop(['alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "lasso_regression_validation_min['model'] = 'Linear Regression Lasso (L1)'\n",
    "lasso_regression_validation_min = lasso_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "ridge_regression_validation_min = ridge_regression_validation_min.drop(['alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "ridge_regression_validation_min['model'] = 'Linear Regression Ridge (L2)'\n",
    "ridge_regression_validation_min = ridge_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "elastic_net_validation_min = elastic_net_validation_min.drop(['alpha', 'l1_ratio','max_iter'], axis=1).iloc[[0]]\n",
    "elastic_net_validation_min['model'] = 'Linear Regression Elastic Net (L1 & L2)'\n",
    "elastic_net_validation_min = elastic_net_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_regression_validation_min = polynomial_regression_validation_min.drop('degree', axis=1).iloc[[0]]\n",
    "polynomial_regression_validation_min['model'] = 'Polynomial Regression'\n",
    "polynomial_regression_validation_min = polynomial_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_lasso_regression_validation_min = polynomial_lasso_regression_validation_min.drop(['degree','alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_lasso_regression_validation_min['model'] = 'Polynomial Regression Lasso (L1)'\n",
    "polynomial_lasso_regression_validation_min = polynomial_lasso_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_ridge_regression_validation_min = polynomial_ridge_regression_validation_min.drop(['degree','alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_ridge_regression_validation_min['model'] = 'Polynomial Regression Ridge (L2)'\n",
    "polynomial_ridge_regression_validation_min = polynomial_ridge_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_elastic_net_regression_validation_min = polynomial_elastic_net_regression_validation_min.drop(['degree', 'alpha', 'l1_ratio', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_elastic_net_regression_validation_min['model'] = 'Polynomial Regression Elastic Net (L1 & L2)'\n",
    "polynomial_elastic_net_regression_validation_min = polynomial_elastic_net_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "tree_regression_validation_min = tree_regression_validation_min.drop('max_depth', axis=1).iloc[[0]]\n",
    "tree_regression_validation_min['model'] = 'Decision Tree Regression'\n",
    "tree_regression_validation_min = tree_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "rf_regression_validation_min = rf_regression_validation_min.drop(['n_estimators', 'max_depth'], axis=1).iloc[[0]]\n",
    "rf_regression_validation_min['model'] = 'Random Forest Regression'\n",
    "rf_regression_validation_min = rf_regression_validation_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Linear Regression</td>\n",
       "      <td>3.992483e-02</td>\n",
       "      <td>458.447042</td>\n",
       "      <td>21.411376</td>\n",
       "      <td>17.039754</td>\n",
       "      <td>8.682542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Linear Regression Lasso (L1)</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Linear Regression Ridge (L2)</td>\n",
       "      <td>3.681042e-02</td>\n",
       "      <td>459.934208</td>\n",
       "      <td>21.446077</td>\n",
       "      <td>17.043389</td>\n",
       "      <td>8.672408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Linear Regression Elastic Net (L1 &amp; L2)</td>\n",
       "      <td>-7.197077e-07</td>\n",
       "      <td>477.511956</td>\n",
       "      <td>21.852047</td>\n",
       "      <td>17.352836</td>\n",
       "      <td>8.678722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Polynomial Regression</td>\n",
       "      <td>6.647668e-02</td>\n",
       "      <td>445.768223</td>\n",
       "      <td>21.113224</td>\n",
       "      <td>16.749939</td>\n",
       "      <td>8.547931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Polynomial Regression Lasso (L1)</td>\n",
       "      <td>1.414802e-02</td>\n",
       "      <td>470.755769</td>\n",
       "      <td>21.696907</td>\n",
       "      <td>17.180595</td>\n",
       "      <td>8.655828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Polynomial Regression Ridge (L2)</td>\n",
       "      <td>-9.685894e+00</td>\n",
       "      <td>5102.638358</td>\n",
       "      <td>71.432754</td>\n",
       "      <td>18.826258</td>\n",
       "      <td>8.348152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Polynomial Regression Elastic Net (L1 &amp; L2)</td>\n",
       "      <td>5.457368e-02</td>\n",
       "      <td>451.452047</td>\n",
       "      <td>21.247401</td>\n",
       "      <td>16.808199</td>\n",
       "      <td>8.630816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Decision Tree Regression</td>\n",
       "      <td>-2.519300e-01</td>\n",
       "      <td>597.811099</td>\n",
       "      <td>24.450176</td>\n",
       "      <td>17.132240</td>\n",
       "      <td>7.054840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Random Forest Regression</td>\n",
       "      <td>3.210335e-01</td>\n",
       "      <td>324.214408</td>\n",
       "      <td>18.005955</td>\n",
       "      <td>13.296330</td>\n",
       "      <td>7.066447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         model      r2_score    mse_score  \\\n",
       "0                            Linear Regression  3.992483e-02   458.447042   \n",
       "1                 Linear Regression Lasso (L1) -7.197077e-07   477.511956   \n",
       "2                 Linear Regression Ridge (L2)  3.681042e-02   459.934208   \n",
       "3      Linear Regression Elastic Net (L1 & L2) -7.197077e-07   477.511956   \n",
       "4                        Polynomial Regression  6.647668e-02   445.768223   \n",
       "5             Polynomial Regression Lasso (L1)  1.414802e-02   470.755769   \n",
       "6             Polynomial Regression Ridge (L2) -9.685894e+00  5102.638358   \n",
       "7  Polynomial Regression Elastic Net (L1 & L2)  5.457368e-02   451.452047   \n",
       "8                     Decision Tree Regression -2.519300e-01   597.811099   \n",
       "9                     Random Forest Regression  3.210335e-01   324.214408   \n",
       "\n",
       "   rmse_score  mae_score  mape_score  \n",
       "0   21.411376  17.039754    8.682542  \n",
       "1   21.852047  17.352836    8.678722  \n",
       "2   21.446077  17.043389    8.672408  \n",
       "3   21.852047  17.352836    8.678722  \n",
       "4   21.113224  16.749939    8.547931  \n",
       "5   21.696907  17.180595    8.655828  \n",
       "6   71.432754  18.826258    8.348152  \n",
       "7   21.247401  16.808199    8.630816  \n",
       "8   24.450176  17.132240    7.054840  \n",
       "9   18.005955  13.296330    7.066447  "
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_regression_results = pd.concat([linear_regression_validation_min, lasso_regression_validation_min, ridge_regression_validation_min,\n",
    "                                         elastic_net_validation_min, polynomial_regression_validation_min, polynomial_lasso_regression_validation_min,\n",
    "                                         polynomial_ridge_regression_validation_min, polynomial_elastic_net_regression_validation_min,\n",
    "                                         tree_regression_validation_min, rf_regression_validation_min])\n",
    "validation_regression_results = validation_regression_results.reset_index(drop=True)\n",
    "validation_regression_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.11.3 Test Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression_test_min['model'] = 'Linear Regression'\n",
    "linear_regression_test_min = linear_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "lasso_regression_test_min = lasso_regression_test_min.drop(['alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "lasso_regression_test_min['model'] = 'Linear Regression Lasso (L1)'\n",
    "lasso_regression_test_min = lasso_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "ridge_regression_test_min = ridge_regression_test_min.drop(['alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "ridge_regression_test_min['model'] = 'Linear Regression Ridge (L2)'\n",
    "ridge_regression_test_min = ridge_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "elastic_net_test_min = elastic_net_test_min.drop(['alpha', 'l1_ratio','max_iter'], axis=1).iloc[[0]]\n",
    "elastic_net_test_min['model'] = 'Linear Regression Elastic Net (L1 & L2)'\n",
    "elastic_net_test_min = elastic_net_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_regression_test_min = polynomial_regression_test_min.drop('degree', axis=1).iloc[[0]]\n",
    "polynomial_regression_test_min['model'] = 'Polynomial Regression'\n",
    "polynomial_regression_test_min = polynomial_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_lasso_regression_test_min = polynomial_lasso_regression_test_min.drop(['degree','alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_lasso_regression_test_min['model'] = 'Polynomial Regression Lasso (L1)'\n",
    "polynomial_lasso_regression_test_min = polynomial_lasso_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_ridge_regression_test_min = polynomial_ridge_regression_test_min.drop(['degree','alpha', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_ridge_regression_test_min['model'] = 'Polynomial Regression Ridge (L2)'\n",
    "polynomial_ridge_regression_test_min = polynomial_ridge_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "polynomial_elastic_net_regression_test_min = polynomial_elastic_net_regression_test_min.drop(['degree', 'alpha', 'l1_ratio', 'max_iter'], axis=1).iloc[[0]]\n",
    "polynomial_elastic_net_regression_test_min['model'] = 'Polynomial Regression Elastic Net (L1 & L2)'\n",
    "polynomial_elastic_net_regression_test_min = polynomial_elastic_net_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "tree_regression_test_min = tree_regression_test_min.drop('max_depth', axis=1).iloc[[0]]\n",
    "tree_regression_test_min['model'] = 'Decision Tree Regression'\n",
    "tree_regression_test_min = tree_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n",
    "\n",
    "rf_regression_test_min = rf_regression_test_min.drop(['n_estimators', 'max_depth'], axis=1).iloc[[0]]\n",
    "rf_regression_test_min['model'] = 'Random Forest Regression'\n",
    "rf_regression_test_min = rf_regression_test_min[['model', 'r2_score', 'mse_score', 'rmse_score', 'mae_score', 'mape_score']]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>mse_score</th>\n",
       "      <th>rmse_score</th>\n",
       "      <th>mae_score</th>\n",
       "      <th>mape_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Linear Regression</td>\n",
       "      <td>0.052317</td>\n",
       "      <td>461.427719</td>\n",
       "      <td>21.480869</td>\n",
       "      <td>17.129965</td>\n",
       "      <td>8.521859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Linear Regression Lasso (L1)</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.714550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Linear Regression Ridge (L2)</td>\n",
       "      <td>0.052310</td>\n",
       "      <td>461.431102</td>\n",
       "      <td>21.480947</td>\n",
       "      <td>17.129678</td>\n",
       "      <td>8.522815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Linear Regression Elastic Net (L1 &amp; L2)</td>\n",
       "      <td>-0.000124</td>\n",
       "      <td>486.961469</td>\n",
       "      <td>22.067203</td>\n",
       "      <td>17.551492</td>\n",
       "      <td>8.714550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Polynomial Regression</td>\n",
       "      <td>-0.261752</td>\n",
       "      <td>614.348098</td>\n",
       "      <td>24.786046</td>\n",
       "      <td>17.178214</td>\n",
       "      <td>7.956229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Polynomial Regression Lasso (L1)</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>486.899198</td>\n",
       "      <td>22.065792</td>\n",
       "      <td>17.550363</td>\n",
       "      <td>8.714456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Polynomial Regression Ridge (L2)</td>\n",
       "      <td>-159.704238</td>\n",
       "      <td>78247.050076</td>\n",
       "      <td>279.726742</td>\n",
       "      <td>21.433575</td>\n",
       "      <td>8.113779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Polynomial Regression Elastic Net (L1 &amp; L2)</td>\n",
       "      <td>-0.169653</td>\n",
       "      <td>569.505007</td>\n",
       "      <td>23.864304</td>\n",
       "      <td>16.998383</td>\n",
       "      <td>8.488322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Decision Tree Regression</td>\n",
       "      <td>-0.192656</td>\n",
       "      <td>580.705338</td>\n",
       "      <td>24.097828</td>\n",
       "      <td>16.961002</td>\n",
       "      <td>6.358549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Random Forest Regression</td>\n",
       "      <td>0.280780</td>\n",
       "      <td>350.188801</td>\n",
       "      <td>18.713332</td>\n",
       "      <td>13.855007</td>\n",
       "      <td>6.514113</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         model    r2_score     mse_score  \\\n",
       "0                            Linear Regression    0.052317    461.427719   \n",
       "1                 Linear Regression Lasso (L1)   -0.000124    486.961469   \n",
       "2                 Linear Regression Ridge (L2)    0.052310    461.431102   \n",
       "3      Linear Regression Elastic Net (L1 & L2)   -0.000124    486.961469   \n",
       "4                        Polynomial Regression   -0.261752    614.348098   \n",
       "5             Polynomial Regression Lasso (L1)    0.000004    486.899198   \n",
       "6             Polynomial Regression Ridge (L2) -159.704238  78247.050076   \n",
       "7  Polynomial Regression Elastic Net (L1 & L2)   -0.169653    569.505007   \n",
       "8                     Decision Tree Regression   -0.192656    580.705338   \n",
       "9                     Random Forest Regression    0.280780    350.188801   \n",
       "\n",
       "   rmse_score  mae_score  mape_score  \n",
       "0   21.480869  17.129965    8.521859  \n",
       "1   22.067203  17.551492    8.714550  \n",
       "2   21.480947  17.129678    8.522815  \n",
       "3   22.067203  17.551492    8.714550  \n",
       "4   24.786046  17.178214    7.956229  \n",
       "5   22.065792  17.550363    8.714456  \n",
       "6  279.726742  21.433575    8.113779  \n",
       "7   23.864304  16.998383    8.488322  \n",
       "8   24.097828  16.961002    6.358549  \n",
       "9   18.713332  13.855007    6.514113  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_regression_results = pd.concat([linear_regression_test_min, lasso_regression_test_min, ridge_regression_test_min,\n",
    "                                         elastic_net_test_min, polynomial_regression_test_min, polynomial_lasso_regression_test_min,\n",
    "                                         polynomial_ridge_regression_test_min, polynomial_elastic_net_regression_test_min,\n",
    "                                         tree_regression_test_min, rf_regression_test_min])\n",
    "test_regression_results = test_regression_results.reset_index(drop=True)\n",
    "test_regression_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.0 CLUSTERING MODELS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   The metric chosen to evaluate model performance was Silhouette Score, since it combines both internal cluster cohesion as well as distance within clusters, thus returning a solid estimate on clusters quality definition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_clustering = pd.read_csv('datasets\\\\clustering\\\\X_dataset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering dataset: rows = 178, columns = 13\n"
     ]
    }
   ],
   "source": [
    "print(f'Clustering dataset: rows = {X_clustering.shape[0]}, columns = {X_clustering.shape[1]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Also important to note that a clustering dataset doesnt have a class variable labeled; instead, the model will aim to group the rows of this dataset into different groups(clusters)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 K-Means Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'n_clusters': np.arange(2, 11).tolist()}\n",
    "\n",
    "n_clusters_list = list()\n",
    "\n",
    "ss_score_list = list()\n",
    "\n",
    "for i in params.get('n_clusters'):\n",
    "    n_clusters_list.append(i)\n",
    "    #Model Define\n",
    "    k_means_clustering = c.KMeans(n_clusters=i, random_state=42)\n",
    "    \n",
    "    #Model Prediction\n",
    "    k_means_labels = k_means_clustering.fit_predict(X_clustering)\n",
    "    \n",
    "    #score\n",
    "    ss_score = mt.silhouette_score(X_clustering, k_means_labels)\n",
    "    ss_score_list.append(ss_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_clusters(k)</th>\n",
       "      <th>silhouette_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0.213219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0.232959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>0.216242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>0.188022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>0.210835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7</td>\n",
       "      <td>0.197552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>8</td>\n",
       "      <td>0.189821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9</td>\n",
       "      <td>0.177528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>10</td>\n",
       "      <td>0.186133</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   n_clusters(k)  silhouette_score\n",
       "0              2          0.213219\n",
       "1              3          0.232959\n",
       "2              4          0.216242\n",
       "3              5          0.188022\n",
       "4              6          0.210835\n",
       "5              7          0.197552\n",
       "6              8          0.189821\n",
       "7              9          0.177528\n",
       "8             10          0.186133"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_means_clustering_df = pd.DataFrame({ 'n_clusters(k)': n_clusters_list, 'silhouette_score': ss_score_list})\n",
    "k_means_clustering_df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_clusters(k)</th>\n",
       "      <th>silhouette_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0.232959</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   n_clusters(k)  silhouette_score\n",
       "1              3          0.232959"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_means_clustering_max = k_means_clustering_df.query('silhouette_score == silhouette_score.max()')\n",
    "k_means_clustering_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Affinity Propagation Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "preference = np.arange(-1, -200, -1)\n",
    "\n",
    "preference_list = list()\n",
    "\n",
    "ss_score_list = list()\n",
    "\n",
    "for i in range(len(preference)):\n",
    "    preference_list.append(i)\n",
    "    #Model Define\n",
    "    ap_clustering = c.AffinityPropagation(preference=preference[i], random_state=42)\n",
    "    \n",
    "    #Model Training\n",
    "    ap_clustering.fit(X_clustering)\n",
    "    \n",
    "    #Model Labeling(Clustering)\n",
    "    ap_labels = ap_clustering.predict(X_clustering)\n",
    "    \n",
    "    #Clustering Scores\n",
    "    ss_score = mt.silhouette_score(X_clustering, ap_labels)\n",
    "    ss_score_list.append(ss_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>preference</th>\n",
       "      <th>silhouette_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.046663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.129858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.148043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0.156879</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0.164493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194</th>\n",
       "      <td>194</td>\n",
       "      <td>0.198860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>195</td>\n",
       "      <td>0.176173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>196</td>\n",
       "      <td>0.195695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>197</td>\n",
       "      <td>0.198159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>198</td>\n",
       "      <td>0.198159</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>199 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     preference  silhouette_score\n",
       "0             0          0.046663\n",
       "1             1          0.129858\n",
       "2             2          0.148043\n",
       "3             3          0.156879\n",
       "4             4          0.164493\n",
       "..          ...               ...\n",
       "194         194          0.198860\n",
       "195         195          0.176173\n",
       "196         196          0.195695\n",
       "197         197          0.198159\n",
       "198         198          0.198159\n",
       "\n",
       "[199 rows x 2 columns]"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ap_clustering_df = pd.DataFrame({ 'preference': preference_list, 'silhouette_score': ss_score_list})\n",
    "ap_clustering_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>preference</th>\n",
       "      <th>silhouette_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>141</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>142</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>143</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>144</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>145</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>148</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>149</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>155</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>156</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>157</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>158</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>175</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>179</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>181</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>182</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>184</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     preference  silhouette_score\n",
       "141         141          0.215123\n",
       "142         142          0.215123\n",
       "143         143          0.215123\n",
       "144         144          0.215123\n",
       "145         145          0.215123\n",
       "148         148          0.215123\n",
       "149         149          0.215123\n",
       "155         155          0.215123\n",
       "156         156          0.215123\n",
       "157         157          0.215123\n",
       "158         158          0.215123\n",
       "175         175          0.215123\n",
       "179         179          0.215123\n",
       "181         181          0.215123\n",
       "182         182          0.215123\n",
       "184         184          0.215123"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ap_clustering_max = ap_clustering_df.query('silhouette_score == silhouette_score.max()')\n",
    "ap_clustering_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Clustering Final Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_means_clustering_max = k_means_clustering_max.drop('n_clusters(k)', axis=1).iloc[[0]]\n",
    "k_means_clustering_max['model'] = 'K-Means Clustering'\n",
    "k_means_clustering_max = k_means_clustering_max[['model', 'silhouette_score']]\n",
    "\n",
    "ap_clustering_max = ap_clustering_max.drop('preference', axis=1).iloc[[0]]\n",
    "ap_clustering_max['model'] = 'Affinity Propagation Clustering'\n",
    "ap_clustering_max = ap_clustering_max[['model', 'silhouette_score']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>silhouette_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>K-Means Clustering</td>\n",
       "      <td>0.232959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Affinity Propagation Clustering</td>\n",
       "      <td>0.215123</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             model  silhouette_score\n",
       "0               K-Means Clustering          0.232959\n",
       "1  Affinity Propagation Clustering          0.215123"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clustering_results = pd.concat([k_means_clustering_max, ap_clustering_max]).reset_index(drop=True)\n",
    "clustering_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   We can see that both models converge according to different parameter setup: The K-Means presented better cluster distinction when using 3 clusters on the dataset, while Affinity Propagation model converged on a higher preference value."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fundamentos_ml",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}