regression_metrics.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align='center'> Performance Metrics for Regression Problems </h1>\n",
    "\n",
    "In this note book, I am going to illsutrate how we can evaluate the ML models deployed for regression task using different kind of evaluation metrics.\n",
    "\n",
    "For illsutareting purpose I have collected the data from kaggle. I will be doing all the analysis over them only. I have attach the link for each dataset, you can also download the same.\n",
    "\n",
    "\n",
    "For the implemetations of these metrics, I am using following tools and frameworks:\n",
    "- Python - as a primary language\n",
    "- Pandas - as an analytical engine for processing the data\n",
    "- numpy - for computation using numpy arrays\n",
    "- matplotlib - for plotting the figures\n",
    "- sklearn - for implememting the metrics\n",
    "- seaborn - for graph plotting \n",
    "\n",
    "Note that I will be implementing all the metrices from scratch.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write some generic code which will be used throughout this notebook.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Imports](#imports)\n",
    "1. [Data Loader](#data-loader)\n",
    "1. [Basic Exploratory Data Analysis (EDA)](#basic-exploratory-data-analysis-eda)\n",
    "1. [Modeling](#modeling)\n",
    "    1. [Train/Test Splits](#traintest-split)\n",
    "    1. [Model Selection](#model-selection)\n",
    "    1. [Model Training](#model-training)\n",
    "1. [Model Evaluation](#model-evaluation)\n",
    "    1. [Mean Squared Error](#mean-squared-error-mse)\n",
    "    1. [Root Mean Squared Error](#root-mean-squared-errorrmse)\n",
    "    1. [Mean Absolute Error](#mean-absolute-error-mae)  \n",
    "    1. [R-squared scores](#r-squared)\n",
    "    1. [Pearson correlation](#Pearson-correlation)\n",
    "    1. [Spearman rank correlation](#Spearman-rank-correlation)\n",
    "    1. [Adjusted R-squared](#adjusted-r-squared)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn\n",
    "import sklearn"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loader\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for loading a csv file from specified path\n",
    "\n",
    "def data_loader(path):\n",
    "    \n",
    "    #use pandas to load the data from path\n",
    "    df = pd.read_csv(path)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<h2 align='center'>Boston Housing Price Prediction</h2>\n",
    "\n",
    "**Problem Statement :**\n",
    "\n",
    "Given the information about locality, crimes and other relevant information about locality of Boston City(USA) predict a range of the  price of the houses.\n",
    "\n",
    "**About Data :** \n",
    "\n",
    "The Boston Housing Dataset is a derived from information collected by the U.S. Census Service concerning housing in the area of Boston MA. The Data is from kaggle and same can be downloaded from <a href=\"https://www.kaggle.com/code/kmalit/bank-customer-churn-prediction\">here</a>.\n",
    "\n",
    "**Columns Info:**\n",
    "\n",
    " The following describes the dataset columns:\n",
    "\n",
    "- **CRIM -** per capita crime rate by town\n",
    "- **ZN -** proportion of residential land zoned for lots over `25,000` sq.ft.\n",
    "- **INDUS -** proportion of non-retail business acres per town.\n",
    "- **CHAS -** Charles River dummy variable (`1` if tract bounds river; `0` otherwise)\n",
    "- **NOX -** nitric oxides concentration (parts per `10` million)\n",
    "- **RM -** average number of rooms per dwelling\n",
    "- **AGE -** proportion of owner-occupied units built prior to `1940`\n",
    "- **DIS -** weighted distances to five Boston employment centres\n",
    "- **RAD -** index of accessibility to radial highways\n",
    "- **TAX -** full-value property-tax rate per `$10,000`\n",
    "- **PTRATIO -** pupil-teacher ratio by town\n",
    "- **B -** `1000` `(Bk - 0.63)^2` where `Bk` is the proportion of blacks by town\n",
    "- **LSTAT -** % lower status of the population\n",
    "- **MEDV -** Median value of owner-occupied homes in `$1000`'s\n",
    "\n",
    "**Variable of Interest Seggregation**\n",
    "\n",
    "For the problem statement and data info we can deduce following:\n",
    "- **Target Variable -** `Median value of owner-occupied homes (MEDV)`\n",
    "- **Independent Variables -** `Rest of the columns (CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, B, LSTAT)`\n",
    "\n",
    "\n",
    "**Algorithms to be used :**\n",
    "\n",
    "Since this is a regression problem I will be using `Linear Regression`, `Polynomial Regression`, `Random Forest Regressor` and `XGBoost Regressor` which are based on Regression Algorithms for modeling pourpose.\n",
    "\n",
    "**Evaluations metrics :**\n",
    "\n",
    "Since this is a regression problem, I will be using following metrics for evaluating output of the predictive models.\n",
    "\n",
    "- Mean Squared Error(MSE)\n",
    "- Root Mean Squared Error(RMSE)\n",
    "- Mean Absolute Error(MAE)\n",
    "- Coefficient of Determinations(R^2)\n",
    "- Adjusted R^2\n",
    "- Huber Loss\n",
    "- Mean Percentage Error(MAE)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Always look at your data`\n",
    "## Load the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have downloaded the data from above link and have stored the same in my local file system. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>MEDV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.09</td>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.9</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE   DIS  RAD  TAX  PTRATIO  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.09    1  296     15.3   \n",
       "\n",
       "       B  LSTAT  MEDV  \n",
       "0  396.9   4.98  24.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#df contains all the data\n",
    "path= \"/Users/ajitkumarsingh/Desktop/Data-Science-Interview-Questions/performance-metrics/data/HousingData.csv\"\n",
    "df = data_loader(path)\n",
    "\n",
    "# show first row\n",
    "df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "columns name : CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, B, LSTAT, MEDV\n",
      "\n",
      "total number of columns : 14\n",
      "\n",
      "Number of rows in data : 506\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#columns name in churn df\n",
    "print(f\"columns name : {', '.join(df.columns.tolist())}\\n\")\n",
    "\n",
    "#total number of columns\n",
    "print(f\"total number of columns : {len(df.columns)}\\n\")\n",
    "\n",
    "#total count of the data\n",
    "print(f\"Number of rows in data : {len(df)}\\n\")\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 14 attributes and 10000 rows in the churn data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       20\n",
       "ZN         20\n",
       "INDUS      20\n",
       "CHAS       20\n",
       "NOX         0\n",
       "RM          0\n",
       "AGE        20\n",
       "DIS         0\n",
       "RAD         0\n",
       "TAX         0\n",
       "PTRATIO     0\n",
       "B           0\n",
       "LSTAT      20\n",
       "MEDV        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of nulls per attributes\n",
    "df.isnull().sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have some null values in the data. The ML models are not directly deal with null values, we need to either drop the rows where null values is present in atleast one of the columns or impute the null values with appropriate values.\n",
    "There are several ways to impute the missing values, Here I am imputing the missing values with the average value of the their entire column. \n",
    "\n",
    "I am using `df.fillna()` to accomplish it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#impute the null values with mean\n",
    "\n",
    "df = df.fillna(df.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       0\n",
       "ZN         0\n",
       "INDUS      0\n",
       "CHAS       0\n",
       "NOX        0\n",
       "RM         0\n",
       "AGE        0\n",
       "DIS        0\n",
       "RAD        0\n",
       "TAX        0\n",
       "PTRATIO    0\n",
       "B          0\n",
       "LSTAT      0\n",
       "MEDV       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of nulls per attributes\n",
    "df.isnull().sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we don't have null values in any of the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       485\n",
       "ZN          27\n",
       "INDUS       77\n",
       "CHAS         3\n",
       "NOX         81\n",
       "RM         446\n",
       "AGE        349\n",
       "DIS        412\n",
       "RAD          9\n",
       "TAX         66\n",
       "PTRATIO     46\n",
       "B          357\n",
       "LSTAT      439\n",
       "MEDV       229\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique values\n",
    "df.nunique()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have some columns like `CRIM` and `LSTAT` with high cardinality. High cardinalty means we have many unique values in that columns and high cardinality might lead to increase in model complexity. It is advisable to reduce the cardinality using `binning` or `discretization` if the we are not so concerned about literal values of that attribute.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       float64\n",
       "ZN         float64\n",
       "INDUS      float64\n",
       "CHAS       float64\n",
       "NOX        float64\n",
       "RM         float64\n",
       "AGE        float64\n",
       "DIS        float64\n",
       "RAD          int64\n",
       "TAX          int64\n",
       "PTRATIO    float64\n",
       "B          float64\n",
       "LSTAT      float64\n",
       "MEDV       float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data types\n",
    "df.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All columns seems to have `numeric` data type which is a good thing because we can directly feed these values to our models and we don't have to worry about encoding and all.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Selection\n",
    "\n",
    "This is the crucial part of a ML modeling. Our main main objective is to predict the price of the houses based on some attributes and it is important to recognise the important features which are key driving factor of the pricing. There are many mathematical and analytical ways to do that, like we can build a correlation matrix and we can find the correlation coefficient to see which columns are more correlated to target variable and also we can figure out correlated columns and can drop any of them to reduce the overall compleity of the model. Another way to use your domain knowledge and analyze each attributes and see if it impacts the target in any way if not drop the feature.\n",
    "\n",
    "\n",
    "For time being I am asssuming all the columns present in `df` are equaly important and are key driving factor in the target value prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>MEDV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.09</td>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.9</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE   DIS  RAD  TAX  PTRATIO  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.09    1  296     15.3   \n",
       "\n",
       "       B  LSTAT  MEDV  \n",
       "0  396.9   4.98  24.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "redundant_features = []\n",
    "\n",
    "#drop redundant features(in this case it is empty)\n",
    "df = df.drop(redundant_features, axis=1)\n",
    "\n",
    "#show 1 row\n",
    "df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Features Extraction\n",
    "\n",
    "Feature extraction is a process in machine learning and data analysis that involves transforming raw data or input features into a set of representative features that are more meaningful and informative for a particular task or problem. It is an important step in the feature engineering process, which involves selecting, transforming, and creating new features from raw data to improve the performance and interpretability of machine learning models.\n",
    "\n",
    "This this problem we noted that we do have some columns which have like high cardinality and we can reduce cardinality using `binning` process.\n",
    "\n",
    "In `binning` method, we convert `numeric` columns to `categorical` columns to reduce overall cardinality of an attribute.\n",
    "\n",
    "Let's say, if the number of distinct values in a column is greater than the `50%` of the total count it means that column has high cardinality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       485\n",
       "ZN          27\n",
       "INDUS       77\n",
       "CHAS         3\n",
       "NOX         81\n",
       "RM         446\n",
       "AGE        349\n",
       "DIS        412\n",
       "RAD          9\n",
       "TAX         66\n",
       "PTRATIO     46\n",
       "B          357\n",
       "LSTAT      439\n",
       "MEDV       229\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique values\n",
    "df.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns with high cardinalities are CRIM, RM, AGE, DIS, B, LSTAT\n"
     ]
    }
   ],
   "source": [
    "#columns with high cardinality\n",
    "total_count = df.shape[0]\n",
    "unique_counts = df.nunique()\n",
    "\n",
    "#filter for high cardinality\n",
    "high_cardinality_columns = unique_counts[unique_counts>0.5*total_count].index.tolist()\n",
    "\n",
    "print(\"Columns with high cardinalities are {}\".format(', '.join(high_cardinality_columns)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discreatize the categorical columns one by one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dicreatization based on quatiles\n",
    "#classify crimes(CRIM) into HIGH, Medium and Low\n",
    "df['CRIM_BINNED'] = pd.qcut(df['CRIM'],q=3,labels=['Low', 'Moderate', \"High\"])\n",
    "\n",
    "#bin the age coliumn into Young, Adult and Old\n",
    "df['AGE_BINNED'] = pd.qcut(df['AGE'],q=3,labels=['Young', 'Adult', \"Old\"])\n",
    "\n",
    "#bin the average number of room per dwelling(RM) into Low, Moderate and High\n",
    "df['RM_BINNED'] = pd.qcut(df['RM'], q=3, labels=['Low', 'Moderate', \"High\"])\n",
    "\n",
    "#bin distance column (DIS) into Very Near, Near, Far and Very Far\n",
    "df['DIS_BINNED'] = pd.qcut(df['DIS'], q=4, labels=['Very Near', 'Near', \"Far\", \"Very Far\"])\n",
    "\n",
    "#bin Black population proportion\n",
    "df['B_BINNED'] = pd.qcut(df['B'], q=2, labels=['Low', 'High'])\n",
    "\n",
    "#bin % of lower status of the popultion (LSTAT)\n",
    "df['LSTAT_BINNED'] = pd.qcut(df['LSTAT'], q=3, labels=['Low', 'Medium','High'])\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** that there might be many permutation and combinations here like what is optimal number of categories a column should be binned. For time being I have used common intution for transforming the above high cardinality columns.\n",
    "\n",
    "Also note that while binning we are replacing the numeric values to some non-numeric data type mainly strings here. We need to encode them before we feed the data to our ML models.\n",
    "\n",
    "There are several techniques for encoding the values into numeric once. Here I am using `One Hot Encoding` method to label the categorical values. In pandas we have `pd.get_dummies(df, columns=column_list)` to get dummies numeric values for non-numeric values. \n",
    "\n",
    "This method in pandas append new columns and the number of these new columns are only dependent on the number of `unique` values in the original columns. For example suppose we have a column name `Gender` and it has two distinct values `Male` and `Female`, this methods will create two new columns namely `Gender_Male` and `Gender_Female` and values of `Gender_Male` will be `0` where `Gender` column is `Female` and `1` if the value is `Male`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove the high catdinality columns now\n",
    "df_binned = df.drop(high_cardinality_columns, axis=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how many unique values we have per attributes after binning high cardinality columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ZN               27\n",
       "INDUS            77\n",
       "CHAS              3\n",
       "NOX              81\n",
       "RAD               9\n",
       "TAX              66\n",
       "PTRATIO          46\n",
       "MEDV            229\n",
       "CRIM_BINNED       3\n",
       "AGE_BINNED        3\n",
       "RM_BINNED         3\n",
       "DIS_BINNED        4\n",
       "B_BINNED          2\n",
       "LSTAT_BINNED      3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of unique values\n",
    "df_binned.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Encode the categorica columns now using One Hot Encoding\n",
    "suffix = '_BINNED'\n",
    "binned_columns = list(filter(lambda x : x[-len(suffix):]==suffix, df.columns))\n",
    "\n",
    "#use pd.get_dummies to encode the above columns\n",
    "df_encoded = pd.get_dummies(df, columns=binned_columns)\n",
    "\n",
    "#now let's see if we have new columns "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX',\n",
       "       'PTRATIO', 'B', 'LSTAT', 'MEDV', 'CRIM_BINNED_Low',\n",
       "       'CRIM_BINNED_Moderate', 'CRIM_BINNED_High', 'AGE_BINNED_Young',\n",
       "       'AGE_BINNED_Adult', 'AGE_BINNED_Old', 'RM_BINNED_Low',\n",
       "       'RM_BINNED_Moderate', 'RM_BINNED_High', 'DIS_BINNED_Very Near',\n",
       "       'DIS_BINNED_Near', 'DIS_BINNED_Far', 'DIS_BINNED_Very Far',\n",
       "       'B_BINNED_Low', 'B_BINNED_High', 'LSTAT_BINNED_Low',\n",
       "       'LSTAT_BINNED_Medium', 'LSTAT_BINNED_High'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#now see the columns name\n",
    "\n",
    "df_encoded.columns"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly `pd.get_dummies` method has appended some new columns in the data.\n",
    "\n",
    "Just make sure all the columns are of `numeric` data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM                    float64\n",
       "ZN                      float64\n",
       "INDUS                   float64\n",
       "CHAS                    float64\n",
       "NOX                     float64\n",
       "RM                      float64\n",
       "AGE                     float64\n",
       "DIS                     float64\n",
       "RAD                       int64\n",
       "TAX                       int64\n",
       "PTRATIO                 float64\n",
       "B                       float64\n",
       "LSTAT                   float64\n",
       "MEDV                    float64\n",
       "CRIM_BINNED_Low           uint8\n",
       "CRIM_BINNED_Moderate      uint8\n",
       "CRIM_BINNED_High          uint8\n",
       "AGE_BINNED_Young          uint8\n",
       "AGE_BINNED_Adult          uint8\n",
       "AGE_BINNED_Old            uint8\n",
       "RM_BINNED_Low             uint8\n",
       "RM_BINNED_Moderate        uint8\n",
       "RM_BINNED_High            uint8\n",
       "DIS_BINNED_Very Near      uint8\n",
       "DIS_BINNED_Near           uint8\n",
       "DIS_BINNED_Far            uint8\n",
       "DIS_BINNED_Very Far       uint8\n",
       "B_BINNED_Low              uint8\n",
       "B_BINNED_High             uint8\n",
       "LSTAT_BINNED_Low          uint8\n",
       "LSTAT_BINNED_Medium       uint8\n",
       "LSTAT_BINNED_High         uint8\n",
       "dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#data types of final df\n",
    "df_encoded.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't have any non numeric data type columns."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "### Train/Test Split\n",
    "\n",
    "For evaluating a model, we need some data to test it on, once training part is done. We usually split the data into two parts i.e train and test. \n",
    "On train data we update the models parameters and on test data we see how the trained model is performing. \n",
    "\n",
    "For spliting the data, I am using `train_test_split` function of module `sklearn.model_selection`. \n",
    "\n",
    "Also we need to segregate the `label` from the rest of the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total count : 506\n",
      "\n",
      "Train Data count : 430\n",
      "\n",
      "Test Data count : 76\n",
      "\n",
      "Train Labels count : 430\n",
      "\n",
      "Test Labels count : 76\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#split the data into test and train in 85:15 ratio and drop the label from the rest of the data\n",
    "\n",
    "train_data, test_data, train_labels, test_labels = train_test_split(\n",
    "        df_encoded.drop(['MEDV'], axis=1), \n",
    "        df_encoded['MEDV'], \n",
    "        test_size=0.15, \n",
    "        random_state=42\n",
    "    )\n",
    "\n",
    "#count after split\n",
    "print(f\"Total count : {len(df_encoded)}\\n\")\n",
    "print(f\"Train Data count : {len(train_data)}\\n\")\n",
    "print(f\"Test Data count : {len(test_data)}\\n\")\n",
    "print(f\"Train Labels count : {len(train_labels)}\\n\")\n",
    "print(f\"Test Labels count : {len(test_labels)}\\n\")\n",
    "\n",
    "# number of rows and labels should match\n",
    "assert len(train_data)==len(train_labels)\n",
    "assert len(test_data)==len(test_labels)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have stored train and test data in `train_data`, `test_data` and train and test labels in `train_labels`, `test_labels`. I will be using them during training and testing time accordingly."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Selection\n",
    "\n",
    "We have numerous models out there to solve same type of poblems. But the interesting part is like we don't know which model will be best fitting our dataset and perform well on the test dataset. We need to make choice here and to do that we need to evaluate these models one by one using some performance metrics.\n",
    "\n",
    "I will be training following models and evaluating their performance on test data\n",
    "\n",
    "- Linear Regressor\n",
    "- Random Forest Regressor\n",
    "- Gradient Boosting Regressor\n",
    "- Neighrest Neighbour Regressor\n",
    "\n",
    "Here, I will be using `sklearn` module of `scikit-learn` library to implement the above mentioned models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import the above models\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.ensemble import GradientBoostingRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a list that contains one instances of each model type\n",
    "\n",
    "model_instances = {\n",
    "          'linear_regressor':LinearRegression(),\n",
    "          'random_forest_regressor': RandomForestRegressor(n_estimators=4, max_depth=4, max_features=4), \n",
    "          'knn_regressor':KNeighborsRegressor(), \n",
    "          'gradient_boosting_regressor':GradientBoostingRegressor(n_estimators=4, max_depth=4)\n",
    "          }\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Training\n",
    "\n",
    "In the above step we have created instances of each model and stored in `model_instances` dict type variable. Now we need to train these instances by feeding the train distribution. \n",
    "\n",
    "Once the training is done save the trained instances in the same dict `model_instances`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#train the models on training set and save the trained model for evaluation\n",
    "\n",
    "for model_name, model in model_instances.items():\n",
    "\n",
    "    #fit the model with train_data and train_labels\n",
    "    model.fit(train_data,train_labels)\n",
    "    \n",
    "    #save the trained model\n",
    "    model_instances[model_name] = model\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metrics For Model Evaluation\n",
    "\n",
    "Now we have trained our models and they are ready to make predictions over test dataset. To make sure the model is predicting meaningfull values not random output we need some metrics to evaluate the output. This is where performance metrics come into picture. \n",
    "\n",
    "Let's implement some of the most widely used evaluation metrics for regresion problems from scratch."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Performance Metrics Implementation\n",
    "\n",
    "##### Mean Squared Error (MSE)\n",
    "\n",
    "It estimated as average squared differences of predicted values and the actual values. For an ideal model `mse` would be equal to `0`. Lower `mse` depicts more accurate predictions. It is expressed in squared units of the target variable\n",
    "\n",
    "Mathematically it is expressed as :\n",
    "\n",
    "$$\n",
    "    MSE = \\frac{1}{n} \\sum_{i=1}^{n} (y_{\\text{pred}, i} - y_{\\text{true}, i})^2\n",
    "$$\n",
    "\n",
    "Note that the above expression is `differential` and hence can we calculate `gradient` for optimizing purposes.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Mean Squared Error (MSE) between true and predicted values.\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- mse (float): Mean Squared Error\n",
    "    \"\"\"\n",
    "\n",
    "def mean_squared_error(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #squared the differences\n",
    "    squared_diff = (y_true - y_pred)**2\n",
    "    #find the avg of squared_diff\n",
    "    mse = np.mean(squared_diff)\n",
    "    return mse\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bounds of mean-squared error**\n",
    "\n",
    "[0, $\\infty$), with 0 the best."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Value encoded by mean-squared error**\n",
    "\n",
    "This measure seeks to summarize the errors made by a regression classifier. The smaller it is, the closer the model's predictions are to the truth. In this sense, it is intuitively like a counterpart to [accuracy](#Accuracy) for classifiers."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Weaknesses of mean-squared error**\n",
    "\n",
    "These values are highly dependent on scale of the output variables, making them very hard to interpret in isolation. One really needs a clear baseline, and scale-independent ways of comparing scores are also needed."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Related to mean-squared error**\n",
    "\n",
    "Scikit-learn implements a variety of closely related measures: [mean absolute error](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error), [mean squared logarithmic error](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_log_error.html#sklearn.metrics.mean_squared_log_error), and [median absolute error](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.median_absolute_error.html#sklearn.metrics.median_absolute_error). I'd say that one should choose among these metrics based on how the output values are scaled and distributed. For instance:\n",
    "\n",
    "* The median absolute error will be less sensitive to outliers than the others.\n",
    "* Mean squared logarithmic error might be more appropriate where the outputs are not strictly speaking linearly increasing."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cautions**\n",
    "\n",
    "- **Sensitive to outliers:** MSE is sensitive to outliers because if a predicted value (`y_pred_i`) is an outlier, the squared difference with the corresponding true value (`y_true_i`) will be larger, potentially resulting in an inflated MSE.\n",
    "\n",
    "- **Imbalanced errors:** MSE treats all errors equally, regardless of their magnitude or direction. This means that overestimation and underestimation errors are weighted equally, even though they may have different implications or costs in practice.\n",
    "\n",
    "- **Not robust to non-Gaussian errors:** MSE assumes that the errors follow a Gaussian (normal) distribution, which may not always be the case in real-world scenarios. If the errors are not normally distributed, MSE may not accurately reflect the model's performance.\n",
    "\n",
    "- **Lack of sensitivity to small errors:** MSE may not capture small errors well, as it squares the differences between predicted and true values. This can lead to a model with good MSE but poor performance in capturing small errors, which may be important in certain applications.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Root Mean Squared Error(RMSE)\n",
    "\n",
    "This is root of mean squared error we talked about. The good thing about this metric is like the unit is in sync with target variable and hence can be used for comparing multiple models at once. Since `RMSE` is expressed in the same units as the dependent variable, making it easy to interpret in the context of the original data\n",
    "\n",
    "The expression of `RMSE` is differentiable just like we have for `MSE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Root Mean Squared Error (RMSE) between true and predicted values.\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- mse (float): Mean Squared Error\n",
    "    \"\"\"\n",
    "\n",
    "def root_mean_squared_error(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #squared the differences\n",
    "    squared_diff = (y_true - y_pred)**2\n",
    "    #find the avg of squared_diff\n",
    "    mse = np.mean(squared_diff)\n",
    "    return np.sqrt(mse)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cautions**\n",
    "\n",
    "- Not robust to outliers\n",
    "- Always positive so not usefull when negative errors make sense.\n",
    "- Bias towards large values\n",
    "- Can be affected by sample size"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Mean Absolute Error (MAE)\n",
    "\n",
    "It is the avearge of absolute difference of predicted and actual values. It is more robust to outliers unlike MSE and RMSE.\n",
    "\n",
    "Mathematically, It can be expressed as:\n",
    "\n",
    "$$\n",
    "\\text{MAE} = \\frac{1}{n} \\sum_{i=1}^{n} |y_{\\text{true}_i} - y_{\\text{pred}_i}|\n",
    "$$\n",
    "\n",
    "Note that it is not differential at zero because of the mod `|.|` function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Mean Absolute Error (MAE) between true and predicted values.\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- mae (float): Mean Absolute Error\n",
    "    \"\"\"\n",
    "\n",
    "def mean_absolute_error(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #squared the differences\n",
    "    absolute_diff = np.absolute(y_true - y_pred)\n",
    "    #find the avg of squared_diff\n",
    "    mae = np.mean(absolute_diff)\n",
    "    return mae"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Coeffiecient Of Determination (R^2)\n",
    "\n",
    "It is a statistical measure used to evaluate the `goodness of fit` of a regression model. It represents the proportion of the variance in the dependent variable that is explained by the independent variables in the model. It is more usefull when we have linear relationship between dependent and independent variables.\n",
    "\n",
    "R-squared ranges from `0` to `1`, with `1` indicating a perfect fit and `0` indicating no fit at all.\n",
    "\n",
    "Mathematially, It can be expressed as:\n",
    "\n",
    "$$\n",
    "\\text{R-squared} = 1 - \\frac{SSR}{SST} = 1.0 - \\frac{\n",
    "      \\sum_{i}^{N} (y_{i} - \\hat{y_{i}})^{2}     \n",
    "    }{\n",
    "       \\sum_{i}^{N} (y_{i} - \\mu)^{2}\n",
    "    }\n",
    "$$\n",
    "\n",
    "Where :\n",
    "\n",
    "- *SSR(Sum of Squared Residuals)* represents the sum of the squared differences between the predicted values and the actual values of the dependent variable.\n",
    "\n",
    "- *SST(Sum of Squared Total)* represents the sum of the squared differences between the actual values of the dependent variable and the mean of the dependent variable.\n",
    "\n",
    "* $R^{2}$ is [closely related to the squared Pearson correlation coefficient](https://en.wikipedia.org/wiki/Coefficient_of_determination#As_squared_correlation_coefficient).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate R-squared value given predicted value and actual label\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- r^2 (float): Coefficient of determination(R^2)\n",
    "    \"\"\"\n",
    "\n",
    "def r2_score(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #sum of squared residuals\n",
    "    SSR = np.sum((y_true - y_pred)**2)\n",
    "    #sum of squared total\n",
    "    y_avg = np.mean(y_true)\n",
    "    SST = np.sum((y_avg-y_true)**2)\n",
    "    return 1 - (float(SSR)/float(SST))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note :** `R-squared` value can be negative due to outliers i.e when `MSE(model) > MSE(Baseline)`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bounds of R-squared scores**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best.\n",
    "\n",
    "**Value encoded by R-squared scores**\n",
    "\n",
    "The numerator in the R$^{2}$ calculation is the sum of errors: \n",
    "\n",
    "$$\n",
    "\\textbf{r2}(y, \\widehat{y}) =\n",
    "    1.0 - \\frac{\n",
    "      \\sum_{i}^{N} (y_{i} - \\hat{y_{i}})^{2}     \n",
    "    }{\n",
    "       \\sum_{i}^{N} (y_{i} - \\mu)^{2}\n",
    "    }\n",
    "$$ \n",
    "\n",
    "In the context of regular linear regression, the model's objective is to minimize the total sum of squares, which is the denominator in the calculation. Thus, $R^{2}$ is based in the ratio between what the model achieved and what its objective was, which is a measure of the goodness of fit of the model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Weaknesses of R-squared scores**\n",
    "\n",
    "For comparative purposes, it's nice that $R^{2}$ is scaled between [0, 1]; as noted above, this lack of scaling makes mean squared error hard to interpret. But this also represents a trade-off: $R^{2}$ doesn't tell us about the magnitude of the errors."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Limitations:\n",
    "\n",
    "- **Lack of information on prediction accuracy:** R-squared does not provide information about the accuracy of the model in making predictions. A high R-squared value does not necessarily guarantee accurate predictions, as the model may still have residual errors or may be overfitting the data.\n",
    "\n",
    "- **Sensitivity to outliers:** R-squared can be affected by outliers in the data. Outliers can disproportionately influence the sum of squared residuals (SSR) component of R-squared, leading to an inflated or deflated R-squared value. \n",
    "\n",
    "- **Inability to determine causality:** R-squared does not provide information about causality. Even if a regression model has a high R-squared value, it does not necessarily mean that the independent variables are causing the observed variation in the dependent variable. There may be other confounding variables or omitted variables that are driving the relationship.\n",
    "\n",
    "- **Limited to linear relationships:** R-squared is a measure of the goodness of fit of a linear regression model, which assumes a linear relationship between the dependent and independent variables. If the relationship is non-linear, R-squared may not accurately reflect the goodness of fit."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Adjusted R-squared \n",
    "\n",
    "`R-squared` suffers from problem that the scores improve on increasing number of predictors even if the additional predictors do not significantly improve the model's ability to explain the variation in the dependent variable. This can result in overfitting, where the model appears to fit the data well but may not generalize well to new data.\n",
    "\n",
    "To overcome the above problem associated with `R-squared`, `Adjusted R-squared` adjusts for the number of predictors in the model, penalizing models with more predictors if the additional predictors do not contribute significantly to the model's ability to explain the variation in the dependent variable.\n",
    "\n",
    "Mathematically It can be expressed as:\n",
    "\n",
    "$$\n",
    "\\text{Adjusted R-squared} = 1 - \\left( \\frac{{(1 - R^2) \\cdot (n - 1)}}{{n - k - 1}} \\right)\n",
    "\n",
    "$$\n",
    "\n",
    "Where:\n",
    "\n",
    "- R-squared (Goodness of Fit): It represents the proportion of variance in the dependent variable explained by the regression model.\n",
    "\n",
    "- n: Total number of observations in the dataset.\n",
    "\n",
    "- k: number of predictors(independent variables) in the regression model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Adjusted R-squared value given predicted and actual label and number of predictors\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "        -- k (integer): Number of predictors\n",
    "\n",
    "    Returns:\n",
    "        -- adjusted r^2 (float): Coefficient of determination(R^2)\n",
    "    \"\"\"\n",
    "\n",
    "def adjusted_r2_score(y_true, y_pred, k):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "\n",
    "    #number of observations\n",
    "    n = len(y_true)\n",
    "    #sum of squared residuals\n",
    "    SSR = np.sum((y_true - y_pred)**2)\n",
    "    #sum of squared total\n",
    "    y_avg = np.mean(y_true)\n",
    "    SST = np.sum((y_avg-y_true)**2)\n",
    "    \n",
    "    #r2_score\n",
    "    r2_score = 1 - (float(SSR)/float(SST))\n",
    "    #adjusted r2 square\n",
    "    r2_score_adj = 1 - float((1-r2_score**2)*(n-1))/float(n - k - 1)\n",
    "    \n",
    "    return r2_score_adj"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** `Adjusted R_squared` value is sensitive to sample size. It means the value may decrease as we increase the sample size even if the model is performing better."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pearson correlation\n",
    "\n",
    "The [Pearson correlation coefficient $\\rho$](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) between two vectors $y$ and $\\widehat{y}$ of dimension $N$ is:\n",
    "\n",
    "$$\n",
    "\\textbf{pearsonr}(y, \\widehat{y}) = \n",
    "\\frac{\n",
    "  \\sum_{i}^{N} (y_{i} - \\mu_{y}) \\cdot (\\widehat{y}_{i} - \\mu_{\\widehat{y}})\n",
    "}{\n",
    "  \\sum_{i}^{N} (y_{i} - \\mu_{y})^{2} \\cdot (\\widehat{y}_{i} - \\mu_{\\widehat{y}})^{2}\n",
    "}\n",
    "$$\n",
    "where $\\mu_{y}$ is the mean of $y$ and $\\mu_{\\widehat{y}}$ is the mean of $\\widehat{y}$.\n",
    "\n",
    "This is implemented as `scipy.stats.pearsonr`, which returns the coefficient and a p-value."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Bounds of Pearson correlations\n",
    "\n",
    "$[-1, 1]$, where $-1$ is a complete negative linear correlation, $+1$ is a complete positive linear correlation, and $0$ is no linear correlation at all."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Weaknesses of Pearson correlation\n",
    "\n",
    "Pearson correlations are highly sensitive to the magnitude of the differences between the gold and predicted values. As a result, they are also very sensitive to outliers."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Related to Pearson correlation\n",
    "\n",
    "* For comparing gold values $y$ and predicted values $\\widehat{y}$, Pearson correlation is equivalent to a linear regression using $\\widehat{y}$ and a bias term to predict $y$. [See this great blog post for details.](https://lindeloev.github.io/tests-as-linear/)\n",
    "\n",
    "* [As noted above](#Related-to-R-squared-scores), there is also a close relationship to R-squared values."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Spearman rank correlation\n",
    "\n",
    "The Spearman rank correlation coefficient between between two vectors $y$ and $\\widehat{y}$ of dimension $N$ is the Pearson coefficient with all of the data mapped to their ranks.\n",
    "\n",
    "\n",
    "It is implemented as `scipy.stats.spearmanr`, which returns the coefficient and a p-value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pd' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb Cell 75\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb#Y150sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m corr_df \u001b[39m=\u001b[39m pd\u001b[39m.\u001b[39mDataFrame({\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb#Y150sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m     \u001b[39m'\u001b[39m\u001b[39my1\u001b[39m\u001b[39m'\u001b[39m: np\u001b[39m.\u001b[39mrandom\u001b[39m.\u001b[39muniform(\u001b[39m-\u001b[39m\u001b[39m10\u001b[39m, \u001b[39m10\u001b[39m, size\u001b[39m=\u001b[39m\u001b[39m1000\u001b[39m),\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb#Y150sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m     \u001b[39m'\u001b[39m\u001b[39my2\u001b[39m\u001b[39m'\u001b[39m: np\u001b[39m.\u001b[39mrandom\u001b[39m.\u001b[39muniform(\u001b[39m-\u001b[39m\u001b[39m10\u001b[39m, \u001b[39m10\u001b[39m, size\u001b[39m=\u001b[39m\u001b[39m1000\u001b[39m)})\n",
      "\u001b[0;31mNameError\u001b[0m: name 'pd' is not defined"
     ]
    }
   ],
   "source": [
    "corr_df = pd.DataFrame({\n",
    "    'y1': np.random.uniform(-10, 10, size=1000),\n",
    "    'y2': np.random.uniform(-10, 10, size=1000)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'scipy' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb Cell 76\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb#Y151sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m scipy\u001b[39m.\u001b[39mstats\u001b[39m.\u001b[39mspearmanr(corr_df[\u001b[39m'\u001b[39m\u001b[39my1\u001b[39m\u001b[39m'\u001b[39m], corr_df[\u001b[39m'\u001b[39m\u001b[39my2\u001b[39m\u001b[39m'\u001b[39m])\n",
      "\u001b[0;31mNameError\u001b[0m: name 'scipy' is not defined"
     ]
    }
   ],
   "source": [
    "scipy.stats.spearmanr(corr_df['y1'], corr_df['y2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'scipy' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb Cell 77\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/regression_metrics.ipynb#Y152sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m scipy\u001b[39m.\u001b[39mstats\u001b[39m.\u001b[39mpearsonr(corr_df[\u001b[39m'\u001b[39m\u001b[39my1\u001b[39m\u001b[39m'\u001b[39m]\u001b[39m.\u001b[39mrank(), corr_df[\u001b[39m'\u001b[39m\u001b[39my2\u001b[39m\u001b[39m'\u001b[39m]\u001b[39m.\u001b[39mrank())\n",
      "\u001b[0;31mNameError\u001b[0m: name 'scipy' is not defined"
     ]
    }
   ],
   "source": [
    "scipy.stats.pearsonr(corr_df['y1'].rank(), corr_df['y2'].rank())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Bounds of Spearman rank correlations\n",
    "\n",
    "$[-1, 1]$, where $-1$ is a complete negative linear correlation, $+1$ is a complete positive linear correlation, and $0$ is no linear correlation at all."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Weaknesses of Spearman rank correlation\n",
    "\n",
    "Unlike Pearson, Spearman is not sensitive to the magnitude of the differences. In fact, it's invariant under all monotonic rescaling, since the values are converted to ranks. This also makes it less sensitive to outliers than Pearson.\n",
    "\n",
    "Of course, these strengths become weaknesses in domains where the raw differences do matter. That said, in most NLU contexts, Spearman will be a good conservative choice for system assessment."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Related to Spearman rank correlation**\n",
    "\n",
    "For comparing gold values $y$ and predicted values $\\widehat{y}$, Pearson correlation is equivalent to a linear regression using $\\textbf{rank}(\\widehat{y})$ and a bias term to predict $\\textbf{rank}(y)$. [See this great blog post for details.](https://lindeloev.github.io/tests-as-linear/)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Prediction\n",
    "\n",
    "Now we have defined the performance metrics, we can use them to evaluate the model predctions.\n",
    "\n",
    "From Train/Test Split Section we have :\n",
    "\n",
    "- `train_data`, `test_data`, `train_labels` and `test_labels` we can use them for models performance evaluations.\n",
    "\n",
    "From Model Training section we have : \n",
    "\n",
    "- trained instances of `Linear Regressor`, `KNN Regressor`, `Random Forest Regressor` and `Gradient Booster Regressor` in `model_instances` dictionary. We need to feed the test data to these instances  to get the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    get predictions from test instances\n",
    "\n",
    "    Args:\n",
    "        trained_model () : trained instance\n",
    "        test_data (pandas dataframe) : test data without label\n",
    "\n",
    "    Returns:\n",
    "        list of predictions against each row in test data\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def make_predictions(trained_model, test_data):\n",
    "\n",
    "    return trained_model.predict(test_data)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#store predictions against each model\n",
    "\n",
    "y_pred_per_model = {}\n",
    "\n",
    "for model_name, model_instance in model_instances.items():\n",
    "    \n",
    "    y_pred_per_model[model_name] = make_predictions(model_instance, test_data) \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have stored model name and it's predictions over test data in `y_pred_per_model` and we have also `test_labels` which is like ground truths."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Evaluation\n",
    "\n",
    "This like the last step we need to perform to compare which classification algorithm is performating relatively better over test distribution. In the above sctions, We are done with implementaion of performance metrics now we can use them to evaluate the trained models.\n",
    "\n",
    "Let's create a report in form of a data frame where headers are `model_name` and the performance metrics we discussed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_squared_error</th>\n",
       "      <th>root_mean_squared_error</th>\n",
       "      <th>mean_absolute_error</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>adjusted_r2_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>linear_regressor</th>\n",
       "      <td>10.948576</td>\n",
       "      <td>3.308863</td>\n",
       "      <td>2.515675</td>\n",
       "      <td>0.832344</td>\n",
       "      <td>0.476358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random_forest_regressor</th>\n",
       "      <td>13.647620</td>\n",
       "      <td>3.694269</td>\n",
       "      <td>2.595294</td>\n",
       "      <td>0.791013</td>\n",
       "      <td>0.361992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>knn_regressor</th>\n",
       "      <td>23.958647</td>\n",
       "      <td>4.894757</td>\n",
       "      <td>3.780789</td>\n",
       "      <td>0.633120</td>\n",
       "      <td>-0.021293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gradient_boosting_regressor</th>\n",
       "      <td>34.745582</td>\n",
       "      <td>5.894538</td>\n",
       "      <td>4.333064</td>\n",
       "      <td>0.467939</td>\n",
       "      <td>-0.331306</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             mean_squared_error  root_mean_squared_error  \\\n",
       "linear_regressor                      10.948576                 3.308863   \n",
       "random_forest_regressor               13.647620                 3.694269   \n",
       "knn_regressor                         23.958647                 4.894757   \n",
       "gradient_boosting_regressor           34.745582                 5.894538   \n",
       "\n",
       "                             mean_absolute_error  r2_score  adjusted_r2_score  \n",
       "linear_regressor                        2.515675  0.832344           0.476358  \n",
       "random_forest_regressor                 2.595294  0.791013           0.361992  \n",
       "knn_regressor                           3.780789  0.633120          -0.021293  \n",
       "gradient_boosting_regressor             4.333064  0.467939          -0.331306  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#test labels\n",
    "y_true = np.array(test_labels)\n",
    "\n",
    "#lets create a dict which contains model_name and the key pair of metric name and its value\n",
    "result_dict = {}\n",
    "\n",
    "#iterate over y_pred_per_model to get model name and corresponding predicted label\n",
    "for model_name, y_pred in y_pred_per_model.items():\n",
    "\n",
    "    #find mse\n",
    "    mse = mean_squared_error(y_true, y_pred)\n",
    "    \n",
    "    #find rmse\n",
    "    rmse = root_mean_squared_error(y_true, y_pred)\n",
    "\n",
    "    #find mae\n",
    "    mae = mean_absolute_error(y_true, y_pred)\n",
    "\n",
    "    #find r2_score\n",
    "    r2_value = r2_score(y_true, y_pred)\n",
    "\n",
    "    #find adjusted r2_score\n",
    "    adjusted_r2_value = adjusted_r2_score(y_true, y_pred, 31)\n",
    "\n",
    "    result_dict[model_name] = {\"mean_squared_error\":mse, \"root_mean_squared_error\":rmse, \"mean_absolute_error\":mae, \"r2_score\":r2_value, \"adjusted_r2_score\":adjusted_r2_value}\n",
    "\n",
    "\n",
    "#convert result_dict into pandas data frame\n",
    "result_df = pd.DataFrame.from_dict(result_dict, orient=\"index\")\n",
    "\n",
    "#disply the result df\n",
    "result_df.head(20)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Observations\n",
    "\n",
    "From above table, we can deduce that `linear regression` is performing best among all the models across all the metrics. 2nd best model is `random forest regressor`.\n",
    "\n",
    "Worst performance is given by `Gradient Boost Regressor` across all the metrics.\n",
    "\n",
    "\n",
    "Note that here I am using the default settings of each model and not doing any hyper parameters tuning. So the above observation is done on default setup of each model and hence can change if we do hyper parameter tunings or optimizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}