slides/1-getting_started_with_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7d8affa-124f-4132-88ba-6729aab0ff2e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/stefmolin/pandas-workshop/main?urlpath=lab/tree/slides/1-getting_started_with_pandas.ipynb) [![View slides in browser](https://img.shields.io/badge/view-slides-orange?logo=reveal.js&logoColor=white)](https://stefaniemolin.com/pandas-workshop/#/section-1)\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c0397e0-4592-48dd-90d9-6a6dc95be2f9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_section-1"
    ]
   },
   "source": [
    "# Section 1: Getting Started With Pandas\n",
    "\n",
    "We will begin by introducing the `Series`, `DataFrame`, and `Index` classes, which are the basic building blocks of the pandas library, and showing how to work with them. By the end of this section, you will be able to create DataFrames and perform operations on them to inspect and filter the data.\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img width=\"40%\" src=\"https://hips.hearstapps.com/elleuk.cdnds.net/16/36/3200x1599/gallery-1473083573-pandas.jpg\" alt=\"Waving panda\" style=\"min-width: 400px\">\n",
    "    <div><small><em><a href=\"https://www.elle.com/uk/life-and-culture/culture/news/a31745/10-ways-to-celebrate-pandas-no-longer-being-endangered/\">Source</a></em></small></div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db14205-a9b1-4f2e-ab73-1735d287c6b8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Learning Path\n",
    "\n",
    "1. **Anatomy of a DataFrame**\n",
    "2. Creating DataFrames\n",
    "3. Inspecting the data\n",
    "4. Extracting subsets\n",
    "5. Calculating summary statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2157875-85e2-4cac-aa9e-7d04e47015a4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_anatomy-of-a-dataframe"
    ]
   },
   "source": [
    "## Anatomy of a DataFrame\n",
    "\n",
    "A **DataFrame** is composed of one or more **Series**. The names of the **Series** form the column names, and the row labels form the **Index**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "071b9893-98aa-4f8e-9dff-b34cf7e56a01",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>GeoLocation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aachen</td>\n",
       "      <td>1</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L5</td>\n",
       "      <td>21</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1880 12:00:00 AM</td>\n",
       "      <td>50.77500</td>\n",
       "      <td>6.08333</td>\n",
       "      <td>(50.775, 6.08333)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aarhus</td>\n",
       "      <td>2</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H6</td>\n",
       "      <td>720</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1951 12:00:00 AM</td>\n",
       "      <td>56.18333</td>\n",
       "      <td>10.23333</td>\n",
       "      <td>(56.18333, 10.23333)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abee</td>\n",
       "      <td>6</td>\n",
       "      <td>Valid</td>\n",
       "      <td>EH4</td>\n",
       "      <td>107000</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1952 12:00:00 AM</td>\n",
       "      <td>54.21667</td>\n",
       "      <td>-113.00000</td>\n",
       "      <td>(54.21667, -113.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Acapulco</td>\n",
       "      <td>10</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Acapulcoite</td>\n",
       "      <td>1914</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1976 12:00:00 AM</td>\n",
       "      <td>16.88333</td>\n",
       "      <td>-99.90000</td>\n",
       "      <td>(16.88333, -99.9)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Achiras</td>\n",
       "      <td>370</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L6</td>\n",
       "      <td>780</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1902 12:00:00 AM</td>\n",
       "      <td>-33.16667</td>\n",
       "      <td>-64.95000</td>\n",
       "      <td>(-33.16667, -64.95)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name   id nametype     recclass  mass (g)  fall  \\\n",
       "0    Aachen    1    Valid           L5        21  Fell   \n",
       "1    Aarhus    2    Valid           H6       720  Fell   \n",
       "2      Abee    6    Valid          EH4    107000  Fell   \n",
       "3  Acapulco   10    Valid  Acapulcoite      1914  Fell   \n",
       "4   Achiras  370    Valid           L6       780  Fell   \n",
       "\n",
       "                     year    reclat    reclong           GeoLocation  \n",
       "0  01/01/1880 12:00:00 AM  50.77500    6.08333     (50.775, 6.08333)  \n",
       "1  01/01/1951 12:00:00 AM  56.18333   10.23333  (56.18333, 10.23333)  \n",
       "2  01/01/1952 12:00:00 AM  54.21667 -113.00000    (54.21667, -113.0)  \n",
       "3  01/01/1976 12:00:00 AM  16.88333  -99.90000     (16.88333, -99.9)  \n",
       "4  01/01/1902 12:00:00 AM -33.16667  -64.95000   (-33.16667, -64.95)  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "meteorites = pd.read_csv('../data/Meteorite_Landings.csv', nrows=5)\n",
    "meteorites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43123c4d-579e-4dc2-981e-ce2e2c361b7e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "*Source: [NASA's Open Data Portal](https://data.nasa.gov/Space-Science/Meteorite-Landings/gh4g-9sfh)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f551654b-3a2d-463d-8f7d-b731729af6ee",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_series"
    ]
   },
   "source": [
    "#### Series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9f080ea9-4e02-49b5-aff0-e33fbca0f70e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      Aachen\n",
       "1      Aarhus\n",
       "2        Abee\n",
       "3    Acapulco\n",
       "4     Achiras\n",
       "Name: name, dtype: object"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "470e1162-6261-4d7d-8628-09138ad675e0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_columns-and-index"
    ]
   },
   "source": [
    "#### Columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8d2f752f-ac30-490b-8885-f4d9e15040aa",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['name', 'id', 'nametype', 'recclass', 'mass (g)', 'fall', 'year',\n",
       "       'reclat', 'reclong', 'GeoLocation'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8db9b93-c19f-4776-bf8a-46d680c5f9ad",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "#### Index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f2527b06-8b03-4543-bbd6-4ecdf44078e2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=5, step=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ec60d2-702e-4dd4-a0c7-7ec5f9f525d7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Learning Path\n",
    "\n",
    "1. Anatomy of a DataFrame\n",
    "2. **Creating DataFrames**\n",
    "3. Inspecting the data\n",
    "4. Extracting subsets\n",
    "5. Calculating summary statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd3f6f0c-b4b2-4d28-9ecb-2c2788253979",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_creating-dataframes"
    ]
   },
   "source": [
    "## Creating DataFrames\n",
    "\n",
    "We can create DataFrames from a variety of sources such as other Python objects, flat files, webscraping, and API requests. Here, we will see just a couple of examples, but be sure to check out [this page](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html) in the documentation for a complete list."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56e2efea-5bb3-4f02-a495-bf2f84557c7b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_read-csv"
    ]
   },
   "source": [
    "### Using a flat file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4eed5d04-aec5-4862-85d0-60b3067711de",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "meteorites = pd.read_csv('../data/Meteorite_Landings.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c12ce67-b1de-40a9-ad22-b39125f6b9cc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_api-request"
    ]
   },
   "source": [
    "### Using data from an API\n",
    "\n",
    "Collect the data from [NASA's Open Data Portal](https://data.nasa.gov/Space-Science/Meteorite-Landings/gh4g-9sfh) using the Socrata Open Data API (SODA) with the `requests` library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f1b681d4-e59f-4037-abcd-e48614691c9f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "response = requests.get(\n",
    "    'https://data.nasa.gov/resource/gh4g-9sfh.json',\n",
    "    params={'$limit': 50_000}\n",
    ")\n",
    "\n",
    "if response.ok:\n",
    "    payload = response.json()\n",
    "else:\n",
    "    print(f'Request was not successful and returned code: {response.status_code}.')\n",
    "    payload = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cceed83a-d7f8-4b38-b02d-20aa0b8e2afb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_create-from-payload"
    ]
   },
   "source": [
    "Create the DataFrame with the resulting payload:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b528ae4f-e7d1-4aa9-acb6-16fea333d6cf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>geolocation</th>\n",
       "      <th>:@computed_region_cbhk_fwbd</th>\n",
       "      <th>:@computed_region_nnqa_25f4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aachen</td>\n",
       "      <td>1</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L5</td>\n",
       "      <td>21</td>\n",
       "      <td>Fell</td>\n",
       "      <td>1880-01-01T00:00:00.000</td>\n",
       "      <td>50.775000</td>\n",
       "      <td>6.083330</td>\n",
       "      <td>{'latitude': '50.775', 'longitude': '6.08333'}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aarhus</td>\n",
       "      <td>2</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H6</td>\n",
       "      <td>720</td>\n",
       "      <td>Fell</td>\n",
       "      <td>1951-01-01T00:00:00.000</td>\n",
       "      <td>56.183330</td>\n",
       "      <td>10.233330</td>\n",
       "      <td>{'latitude': '56.18333', 'longitude': '10.23333'}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abee</td>\n",
       "      <td>6</td>\n",
       "      <td>Valid</td>\n",
       "      <td>EH4</td>\n",
       "      <td>107000</td>\n",
       "      <td>Fell</td>\n",
       "      <td>1952-01-01T00:00:00.000</td>\n",
       "      <td>54.216670</td>\n",
       "      <td>-113.000000</td>\n",
       "      <td>{'latitude': '54.21667', 'longitude': '-113.0'}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     name id nametype recclass    mass  fall                     year  \\\n",
       "0  Aachen  1    Valid       L5      21  Fell  1880-01-01T00:00:00.000   \n",
       "1  Aarhus  2    Valid       H6     720  Fell  1951-01-01T00:00:00.000   \n",
       "2    Abee  6    Valid      EH4  107000  Fell  1952-01-01T00:00:00.000   \n",
       "\n",
       "      reclat      reclong                                        geolocation  \\\n",
       "0  50.775000     6.083330     {'latitude': '50.775', 'longitude': '6.08333'}   \n",
       "1  56.183330    10.233330  {'latitude': '56.18333', 'longitude': '10.23333'}   \n",
       "2  54.216670  -113.000000    {'latitude': '54.21667', 'longitude': '-113.0'}   \n",
       "\n",
       "  :@computed_region_cbhk_fwbd :@computed_region_nnqa_25f4  \n",
       "0                         NaN                         NaN  \n",
       "1                         NaN                         NaN  \n",
       "2                         NaN                         NaN  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(payload)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20c5ca3a-bf32-48cc-8eb5-15fd678092e1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "*Tip: `df.to_csv('data.csv')` writes this data to a new file called `data.csv`.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd04b7e1-c083-44a7-988a-8c7c0ea15032",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Learning Path\n",
    "\n",
    "1. Anatomy of a DataFrame\n",
    "2. Creating DataFrames\n",
    "3. **Inspecting the data**\n",
    "4. Extracting subsets\n",
    "5. Calculating summary statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4364557-213a-4605-a8e9-d56d575e15db",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_inspecting-the-data"
    ]
   },
   "source": [
    "## Inspecting the data\n",
    "Now that we have some data, we need to perform an initial inspection of it. This gives us information on what the data looks like, how many rows/columns there are, and how much data we have. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9cfcc92-e4b0-4dc8-91ed-4d097132f7be",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Let's inspect the `meteorites` data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c56fc4d1-805d-46ff-8ebb-ba00f2df713b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_rows-and-columns"
    ]
   },
   "source": [
    "#### How many rows and columns are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "24268ca3-aa52-491b-bc02-ed0a4d85be6f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45716, 10)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9113617e-b071-467c-afda-9813c4c60801",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "#### What are the column names?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eda374e6-8860-45cd-9625-1893b42c3518",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['name', 'id', 'nametype', 'recclass', 'mass (g)', 'fall', 'year',\n",
       "       'reclat', 'reclong', 'GeoLocation'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e77d9d1a-6c64-4554-be7f-f0e0ecea556e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_dtypes"
    ]
   },
   "source": [
    "#### What type of data does each column currently hold?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ade5a00e-5056-4fc2-bd13-55eaa5cf0328",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name            object\n",
       "id               int64\n",
       "nametype        object\n",
       "recclass        object\n",
       "mass (g)       float64\n",
       "fall            object\n",
       "year            object\n",
       "reclat         float64\n",
       "reclong        float64\n",
       "GeoLocation     object\n",
       "dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d878656-ee43-41ee-9d2a-8663f6b548de",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_head"
    ]
   },
   "source": [
    "#### What does the data look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5a04730d-0470-4038-82c1-26441299e3c2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>GeoLocation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aachen</td>\n",
       "      <td>1</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1880 12:00:00 AM</td>\n",
       "      <td>50.77500</td>\n",
       "      <td>6.08333</td>\n",
       "      <td>(50.775, 6.08333)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aarhus</td>\n",
       "      <td>2</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H6</td>\n",
       "      <td>720.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1951 12:00:00 AM</td>\n",
       "      <td>56.18333</td>\n",
       "      <td>10.23333</td>\n",
       "      <td>(56.18333, 10.23333)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abee</td>\n",
       "      <td>6</td>\n",
       "      <td>Valid</td>\n",
       "      <td>EH4</td>\n",
       "      <td>107000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1952 12:00:00 AM</td>\n",
       "      <td>54.21667</td>\n",
       "      <td>-113.00000</td>\n",
       "      <td>(54.21667, -113.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Acapulco</td>\n",
       "      <td>10</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Acapulcoite</td>\n",
       "      <td>1914.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1976 12:00:00 AM</td>\n",
       "      <td>16.88333</td>\n",
       "      <td>-99.90000</td>\n",
       "      <td>(16.88333, -99.9)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Achiras</td>\n",
       "      <td>370</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L6</td>\n",
       "      <td>780.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1902 12:00:00 AM</td>\n",
       "      <td>-33.16667</td>\n",
       "      <td>-64.95000</td>\n",
       "      <td>(-33.16667, -64.95)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name   id nametype     recclass  mass (g)  fall  \\\n",
       "0    Aachen    1    Valid           L5      21.0  Fell   \n",
       "1    Aarhus    2    Valid           H6     720.0  Fell   \n",
       "2      Abee    6    Valid          EH4  107000.0  Fell   \n",
       "3  Acapulco   10    Valid  Acapulcoite    1914.0  Fell   \n",
       "4   Achiras  370    Valid           L6     780.0  Fell   \n",
       "\n",
       "                     year    reclat    reclong           GeoLocation  \n",
       "0  01/01/1880 12:00:00 AM  50.77500    6.08333     (50.775, 6.08333)  \n",
       "1  01/01/1951 12:00:00 AM  56.18333   10.23333  (56.18333, 10.23333)  \n",
       "2  01/01/1952 12:00:00 AM  54.21667 -113.00000    (54.21667, -113.0)  \n",
       "3  01/01/1976 12:00:00 AM  16.88333  -99.90000     (16.88333, -99.9)  \n",
       "4  01/01/1902 12:00:00 AM -33.16667  -64.95000   (-33.16667, -64.95)  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08c22017-6e52-4ab4-bc90-8f955ef2be0a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_tail"
    ]
   },
   "source": [
    "Sometimes there may be extraneous data at the end of the file, so checking the bottom few rows is also important:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "94e84ed2-4022-4a65-9e4e-865b8bd4b04b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>GeoLocation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>45711</th>\n",
       "      <td>Zillah 002</td>\n",
       "      <td>31356</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Eucrite</td>\n",
       "      <td>172.0</td>\n",
       "      <td>Found</td>\n",
       "      <td>01/01/1990 12:00:00 AM</td>\n",
       "      <td>29.03700</td>\n",
       "      <td>17.01850</td>\n",
       "      <td>(29.037, 17.0185)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45712</th>\n",
       "      <td>Zinder</td>\n",
       "      <td>30409</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Pallasite, ungrouped</td>\n",
       "      <td>46.0</td>\n",
       "      <td>Found</td>\n",
       "      <td>01/01/1999 12:00:00 AM</td>\n",
       "      <td>13.78333</td>\n",
       "      <td>8.96667</td>\n",
       "      <td>(13.78333, 8.96667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45713</th>\n",
       "      <td>Zlin</td>\n",
       "      <td>30410</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H4</td>\n",
       "      <td>3.3</td>\n",
       "      <td>Found</td>\n",
       "      <td>01/01/1939 12:00:00 AM</td>\n",
       "      <td>49.25000</td>\n",
       "      <td>17.66667</td>\n",
       "      <td>(49.25, 17.66667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45714</th>\n",
       "      <td>Zubkovsky</td>\n",
       "      <td>31357</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L6</td>\n",
       "      <td>2167.0</td>\n",
       "      <td>Found</td>\n",
       "      <td>01/01/2003 12:00:00 AM</td>\n",
       "      <td>49.78917</td>\n",
       "      <td>41.50460</td>\n",
       "      <td>(49.78917, 41.5046)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45715</th>\n",
       "      <td>Zulu Queen</td>\n",
       "      <td>30414</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L3.7</td>\n",
       "      <td>200.0</td>\n",
       "      <td>Found</td>\n",
       "      <td>01/01/1976 12:00:00 AM</td>\n",
       "      <td>33.98333</td>\n",
       "      <td>-115.68333</td>\n",
       "      <td>(33.98333, -115.68333)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name     id nametype              recclass  mass (g)   fall  \\\n",
       "45711  Zillah 002  31356    Valid               Eucrite     172.0  Found   \n",
       "45712      Zinder  30409    Valid  Pallasite, ungrouped      46.0  Found   \n",
       "45713        Zlin  30410    Valid                    H4       3.3  Found   \n",
       "45714   Zubkovsky  31357    Valid                    L6    2167.0  Found   \n",
       "45715  Zulu Queen  30414    Valid                  L3.7     200.0  Found   \n",
       "\n",
       "                         year    reclat    reclong             GeoLocation  \n",
       "45711  01/01/1990 12:00:00 AM  29.03700   17.01850       (29.037, 17.0185)  \n",
       "45712  01/01/1999 12:00:00 AM  13.78333    8.96667     (13.78333, 8.96667)  \n",
       "45713  01/01/1939 12:00:00 AM  49.25000   17.66667       (49.25, 17.66667)  \n",
       "45714  01/01/2003 12:00:00 AM  49.78917   41.50460     (49.78917, 41.5046)  \n",
       "45715  01/01/1976 12:00:00 AM  33.98333 -115.68333  (33.98333, -115.68333)  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1923d7f7-8974-475f-a7cf-eb2067d3d35d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_info-method"
    ]
   },
   "source": [
    "#### Get some information about the DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "befbd207-284a-4c4f-8617-ff155989ad5f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 45716 entries, 0 to 45715\n",
      "Data columns (total 10 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   name         45716 non-null  object \n",
      " 1   id           45716 non-null  int64  \n",
      " 2   nametype     45716 non-null  object \n",
      " 3   recclass     45716 non-null  object \n",
      " 4   mass (g)     45585 non-null  float64\n",
      " 5   fall         45716 non-null  object \n",
      " 6   year         45425 non-null  object \n",
      " 7   reclat       38401 non-null  float64\n",
      " 8   reclong      38401 non-null  float64\n",
      " 9   GeoLocation  38401 non-null  object \n",
      "dtypes: float64(3), int64(1), object(6)\n",
      "memory usage: 3.5+ MB\n"
     ]
    }
   ],
   "source": [
    "meteorites.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a56f10e-6c16-48ac-a55c-99e63f8ca2e3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_exercises-section-1-part-1"
    ]
   },
   "source": [
    "### Exercise 1.1\n",
    "\n",
    "##### Create a DataFrame by reading in the `2019_Yellow_Taxi_Trip_Data.csv` file. Examine the first 5 rows.\n",
    "\n",
    "### Exercise 1.2\n",
    "\n",
    "##### Find the dimensions (number of rows and number of columns) in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "901a1bc8-ce7e-42cb-a9c7-3f156569040a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_solutions-section-1-part-1"
    ]
   },
   "source": [
    "### Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6f2da22-9093-4442-b96d-6273c0f40e15",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_solution-1-1"
    ]
   },
   "source": [
    "##### Exercise 1.1 &ndash; Create a DataFrame by reading in the `2019_Yellow_Taxi_Trip_Data.csv` file. Examine the first 5 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bc548d6b-9a86-4acf-9860-24b05f9b842e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vendorid</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>ratecodeid</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>pulocationid</th>\n",
       "      <th>dolocationid</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>2019-10-23T16:39:42.000</td>\n",
       "      <td>2019-10-23T17:14:10.000</td>\n",
       "      <td>1</td>\n",
       "      <td>7.93</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>138</td>\n",
       "      <td>170</td>\n",
       "      <td>1</td>\n",
       "      <td>29.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>7.98</td>\n",
       "      <td>6.12</td>\n",
       "      <td>0.3</td>\n",
       "      <td>47.90</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-10-23T16:32:08.000</td>\n",
       "      <td>2019-10-23T16:45:26.000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.00</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>11</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>10.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>12.30</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2019-10-23T16:08:44.000</td>\n",
       "      <td>2019-10-23T16:21:11.000</td>\n",
       "      <td>1</td>\n",
       "      <td>1.36</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>163</td>\n",
       "      <td>162</td>\n",
       "      <td>1</td>\n",
       "      <td>9.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>15.80</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2019-10-23T16:22:44.000</td>\n",
       "      <td>2019-10-23T16:43:26.000</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>170</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>4.32</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>21.62</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>2019-10-23T16:45:11.000</td>\n",
       "      <td>2019-10-23T16:58:49.000</td>\n",
       "      <td>1</td>\n",
       "      <td>1.96</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>163</td>\n",
       "      <td>236</td>\n",
       "      <td>1</td>\n",
       "      <td>10.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>15.30</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   vendorid     tpep_pickup_datetime    tpep_dropoff_datetime  \\\n",
       "0         2  2019-10-23T16:39:42.000  2019-10-23T17:14:10.000   \n",
       "1         1  2019-10-23T16:32:08.000  2019-10-23T16:45:26.000   \n",
       "2         2  2019-10-23T16:08:44.000  2019-10-23T16:21:11.000   \n",
       "3         2  2019-10-23T16:22:44.000  2019-10-23T16:43:26.000   \n",
       "4         2  2019-10-23T16:45:11.000  2019-10-23T16:58:49.000   \n",
       "\n",
       "   passenger_count  trip_distance  ratecodeid store_and_fwd_flag  \\\n",
       "0                1           7.93           1                  N   \n",
       "1                1           2.00           1                  N   \n",
       "2                1           1.36           1                  N   \n",
       "3                1           1.00           1                  N   \n",
       "4                1           1.96           1                  N   \n",
       "\n",
       "   pulocationid  dolocationid  payment_type  fare_amount  extra  mta_tax  \\\n",
       "0           138           170             1         29.5    1.0      0.5   \n",
       "1            11            26             1         10.5    1.0      0.5   \n",
       "2           163           162             1          9.5    1.0      0.5   \n",
       "3           170           163             1         13.0    1.0      0.5   \n",
       "4           163           236             1         10.5    1.0      0.5   \n",
       "\n",
       "   tip_amount  tolls_amount  improvement_surcharge  total_amount  \\\n",
       "0        7.98          6.12                    0.3         47.90   \n",
       "1        0.00          0.00                    0.3         12.30   \n",
       "2        2.00          0.00                    0.3         15.80   \n",
       "3        4.32          0.00                    0.3         21.62   \n",
       "4        0.50          0.00                    0.3         15.30   \n",
       "\n",
       "   congestion_surcharge  \n",
       "0                   2.5  \n",
       "1                   0.0  \n",
       "2                   2.5  \n",
       "3                   2.5  \n",
       "4                   2.5  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "taxis = pd.read_csv('../data/2019_Yellow_Taxi_Trip_Data.csv')\n",
    "taxis.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e93f4eec-fd42-4737-b0f0-5006965bb25c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "*Source: [NYC Open Data](https://data.cityofnewyork.us/Transportation/2019-Yellow-Taxi-Trip-Data/2upf-qytp) collected via [SODA](https://dev.socrata.com/foundry/data.cityofnewyork.us/2upf-qytp).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c2b540a-8110-4692-9776-2c3e5b8a5e8d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_solution-1-2"
    ]
   },
   "source": [
    "##### Exercise 1.2 &ndash; Find the dimensions (number of rows and number of columns) in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "585301c3-97e4-4a8e-b899-d0bfb40ece60",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 18)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxis.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28e0552f-4323-4370-91b0-3870f587bcc8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Learning Path\n",
    "\n",
    "1. Anatomy of a DataFrame\n",
    "2. Creating DataFrames\n",
    "3. Inspecting the data\n",
    "4. **Extracting subsets**\n",
    "5. Calculating summary statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a348133a-b5a2-4e5b-b680-305c16dca0f3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_extracting-subsets"
    ]
   },
   "source": [
    "## Extracting subsets\n",
    "\n",
    "A crucial part of working with DataFrames is extracting subsets of the data: finding rows that meet a certain set of criteria, isolating columns/rows of interest, etc. After narrowing down our data, we are closer to discovering insights. This section will be the backbone of many analysis tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51afc2d0-aec1-4e92-a329-db7562189a54",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_selecting-columns"
    ]
   },
   "source": [
    "#### Selecting columns\n",
    "\n",
    "We can select columns as attributes if their names would be valid Python variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b033c5d3-d7e1-4fa8-9e29-929ba1fe5403",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0            Aachen\n",
       "1            Aarhus\n",
       "2              Abee\n",
       "3          Acapulco\n",
       "4           Achiras\n",
       "            ...    \n",
       "45711    Zillah 002\n",
       "45712        Zinder\n",
       "45713          Zlin\n",
       "45714     Zubkovsky\n",
       "45715    Zulu Queen\n",
       "Name: name, Length: 45716, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eae393b7-0e4f-4bc8-a0a8-1cb2e744036c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "If they aren't, we have to select them as keys. However, we can select multiple columns at once this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8d7ece3d-fb40-45af-9d6b-ee7218b3a27e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>mass (g)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aachen</td>\n",
       "      <td>21.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aarhus</td>\n",
       "      <td>720.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Abee</td>\n",
       "      <td>107000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Acapulco</td>\n",
       "      <td>1914.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Achiras</td>\n",
       "      <td>780.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45711</th>\n",
       "      <td>Zillah 002</td>\n",
       "      <td>172.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45712</th>\n",
       "      <td>Zinder</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45713</th>\n",
       "      <td>Zlin</td>\n",
       "      <td>3.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45714</th>\n",
       "      <td>Zubkovsky</td>\n",
       "      <td>2167.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45715</th>\n",
       "      <td>Zulu Queen</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>45716 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             name  mass (g)\n",
       "0          Aachen      21.0\n",
       "1          Aarhus     720.0\n",
       "2            Abee  107000.0\n",
       "3        Acapulco    1914.0\n",
       "4         Achiras     780.0\n",
       "...           ...       ...\n",
       "45711  Zillah 002     172.0\n",
       "45712      Zinder      46.0\n",
       "45713        Zlin       3.3\n",
       "45714   Zubkovsky    2167.0\n",
       "45715  Zulu Queen     200.0\n",
       "\n",
       "[45716 rows x 2 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites[['name', 'mass (g)']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13e1ad0e-11af-4f01-9ddd-b1cd4dc4337e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_selecting-rows"
    ]
   },
   "source": [
    "#### Selecting rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cd449e0c-17e6-47b2-b7f9-512320090df1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>GeoLocation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>Benton</td>\n",
       "      <td>5026</td>\n",
       "      <td>Valid</td>\n",
       "      <td>LL6</td>\n",
       "      <td>2840.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1949 12:00:00 AM</td>\n",
       "      <td>45.95000</td>\n",
       "      <td>-67.55000</td>\n",
       "      <td>(45.95, -67.55)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>Berduc</td>\n",
       "      <td>48975</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L6</td>\n",
       "      <td>270.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/2008 12:00:00 AM</td>\n",
       "      <td>-31.91000</td>\n",
       "      <td>-58.32833</td>\n",
       "      <td>(-31.91, -58.32833)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>Béréba</td>\n",
       "      <td>5028</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Eucrite-mmict</td>\n",
       "      <td>18000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1924 12:00:00 AM</td>\n",
       "      <td>11.65000</td>\n",
       "      <td>-3.65000</td>\n",
       "      <td>(11.65, -3.65)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>Berlanguillas</td>\n",
       "      <td>5029</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L6</td>\n",
       "      <td>1440.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1811 12:00:00 AM</td>\n",
       "      <td>41.68333</td>\n",
       "      <td>-3.80000</td>\n",
       "      <td>(41.68333, -3.8)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name     id nametype       recclass  mass (g)  fall  \\\n",
       "100         Benton   5026    Valid            LL6    2840.0  Fell   \n",
       "101         Berduc  48975    Valid             L6     270.0  Fell   \n",
       "102         Béréba   5028    Valid  Eucrite-mmict   18000.0  Fell   \n",
       "103  Berlanguillas   5029    Valid             L6    1440.0  Fell   \n",
       "\n",
       "                       year    reclat   reclong          GeoLocation  \n",
       "100  01/01/1949 12:00:00 AM  45.95000 -67.55000      (45.95, -67.55)  \n",
       "101  01/01/2008 12:00:00 AM -31.91000 -58.32833  (-31.91, -58.32833)  \n",
       "102  01/01/1924 12:00:00 AM  11.65000  -3.65000       (11.65, -3.65)  \n",
       "103  01/01/1811 12:00:00 AM  41.68333  -3.80000     (41.68333, -3.8)  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites[100:104]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c8bf6a3-bcc2-448f-a6d4-44b5b01293e5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_indexing"
    ]
   },
   "source": [
    "#### Indexing\n",
    "\n",
    "We use `iloc[]` to select rows and columns by their position:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4ff42eae-fa1d-4936-bb13-a020281b0728",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>Benton</td>\n",
       "      <td>LL6</td>\n",
       "      <td>2840.0</td>\n",
       "      <td>01/01/1949 12:00:00 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>Berduc</td>\n",
       "      <td>L6</td>\n",
       "      <td>270.0</td>\n",
       "      <td>01/01/2008 12:00:00 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>Béréba</td>\n",
       "      <td>Eucrite-mmict</td>\n",
       "      <td>18000.0</td>\n",
       "      <td>01/01/1924 12:00:00 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>Berlanguillas</td>\n",
       "      <td>L6</td>\n",
       "      <td>1440.0</td>\n",
       "      <td>01/01/1811 12:00:00 AM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name       recclass  mass (g)                    year\n",
       "100         Benton            LL6    2840.0  01/01/1949 12:00:00 AM\n",
       "101         Berduc             L6     270.0  01/01/2008 12:00:00 AM\n",
       "102         Béréba  Eucrite-mmict   18000.0  01/01/1924 12:00:00 AM\n",
       "103  Berlanguillas             L6    1440.0  01/01/1811 12:00:00 AM"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.iloc[100:104, [0, 3, 4, 6]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c9690c7-1f45-450c-ba8e-0b70fc1e3e6a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We use `loc[]` to select by name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bba2c44a-aea5-4482-b087-673104aa9b32",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>2840.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1949 12:00:00 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>270.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/2008 12:00:00 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>18000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1924 12:00:00 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>1440.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1811 12:00:00 AM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>960.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/2004 12:00:00 AM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     mass (g)  fall                    year\n",
       "100    2840.0  Fell  01/01/1949 12:00:00 AM\n",
       "101     270.0  Fell  01/01/2008 12:00:00 AM\n",
       "102   18000.0  Fell  01/01/1924 12:00:00 AM\n",
       "103    1440.0  Fell  01/01/1811 12:00:00 AM\n",
       "104     960.0  Fell  01/01/2004 12:00:00 AM"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.loc[100:104, 'mass (g)':'year']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "177e4fc3-9cd5-4e61-8ac0-f024a8cd15a3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_filtering-with-boolean-masks"
    ]
   },
   "source": [
    "#### Filtering with Boolean masks\n",
    "\n",
    "A **Boolean mask** is a array-like structure of Boolean values &ndash; it's a way to specify which rows/columns we want to select (`True`) and which we don't (`False`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40dcbfaf-acd3-4ca3-8582-07d9eedafc3d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Here's an example of a Boolean mask for meteorites weighing more than 50 grams that were found on Earth (i.e., they were not observed falling):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f0dcb663-dddc-487e-83e5-b41b4fdaa903",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        False\n",
       "1        False\n",
       "2        False\n",
       "3        False\n",
       "4        False\n",
       "         ...  \n",
       "45711     True\n",
       "45712    False\n",
       "45713    False\n",
       "45714     True\n",
       "45715     True\n",
       "Length: 45716, dtype: bool"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(meteorites['mass (g)'] > 50) & (meteorites.fall == 'Found')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5eb55fd-5d09-4da8-be4b-f588b58943d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "**Important**: Take note of the syntax here. We surround each condition with parentheses, and we use bitwise operators (`&`, `|`, `~`) instead of logical operators (`and`, `or`, `not`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57f9cff-2153-493d-9023-d6395e50cbe0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "We can use a Boolean mask to select the subset of meteorites weighing more than 1 million grams (1,000 kilograms or roughly 2,205 pounds) that were observed falling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "18294622-156a-426f-b6bf-674315b74bf2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>GeoLocation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Allende</td>\n",
       "      <td>2278</td>\n",
       "      <td>Valid</td>\n",
       "      <td>CV3</td>\n",
       "      <td>2000000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1969 12:00:00 AM</td>\n",
       "      <td>26.96667</td>\n",
       "      <td>-105.31667</td>\n",
       "      <td>(26.96667, -105.31667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>419</th>\n",
       "      <td>Jilin</td>\n",
       "      <td>12171</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H5</td>\n",
       "      <td>4000000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1976 12:00:00 AM</td>\n",
       "      <td>44.05000</td>\n",
       "      <td>126.16667</td>\n",
       "      <td>(44.05, 126.16667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>506</th>\n",
       "      <td>Kunya-Urgench</td>\n",
       "      <td>12379</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H5</td>\n",
       "      <td>1100000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1998 12:00:00 AM</td>\n",
       "      <td>42.25000</td>\n",
       "      <td>59.20000</td>\n",
       "      <td>(42.25, 59.2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>707</th>\n",
       "      <td>Norton County</td>\n",
       "      <td>17922</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Aubrite</td>\n",
       "      <td>1100000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1948 12:00:00 AM</td>\n",
       "      <td>39.68333</td>\n",
       "      <td>-99.86667</td>\n",
       "      <td>(39.68333, -99.86667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>920</th>\n",
       "      <td>Sikhote-Alin</td>\n",
       "      <td>23593</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Iron, IIAB</td>\n",
       "      <td>23000000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1947 12:00:00 AM</td>\n",
       "      <td>46.16000</td>\n",
       "      <td>134.65333</td>\n",
       "      <td>(46.16, 134.65333)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name     id nametype    recclass    mass (g)  fall  \\\n",
       "29         Allende   2278    Valid         CV3   2000000.0  Fell   \n",
       "419          Jilin  12171    Valid          H5   4000000.0  Fell   \n",
       "506  Kunya-Urgench  12379    Valid          H5   1100000.0  Fell   \n",
       "707  Norton County  17922    Valid     Aubrite   1100000.0  Fell   \n",
       "920   Sikhote-Alin  23593    Valid  Iron, IIAB  23000000.0  Fell   \n",
       "\n",
       "                       year    reclat    reclong             GeoLocation  \n",
       "29   01/01/1969 12:00:00 AM  26.96667 -105.31667  (26.96667, -105.31667)  \n",
       "419  01/01/1976 12:00:00 AM  44.05000  126.16667      (44.05, 126.16667)  \n",
       "506  01/01/1998 12:00:00 AM  42.25000   59.20000           (42.25, 59.2)  \n",
       "707  01/01/1948 12:00:00 AM  39.68333  -99.86667   (39.68333, -99.86667)  \n",
       "920  01/01/1947 12:00:00 AM  46.16000  134.65333      (46.16, 134.65333)  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites[(meteorites['mass (g)'] > 1e6) & (meteorites.fall == 'Fell')]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8c71775-3657-4b5b-b68b-1519829a76e1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "*Tip: Boolean masks can be used with `loc[]` and `iloc[]`.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce56faca-d470-4a11-9b80-7b942951a9aa",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_query-method"
    ]
   },
   "source": [
    "An alternative to this is the `query()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2ef48350-5dc0-49b2-93e1-b02f715d18a3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>GeoLocation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Allende</td>\n",
       "      <td>2278</td>\n",
       "      <td>Valid</td>\n",
       "      <td>CV3</td>\n",
       "      <td>2000000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1969 12:00:00 AM</td>\n",
       "      <td>26.96667</td>\n",
       "      <td>-105.31667</td>\n",
       "      <td>(26.96667, -105.31667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>419</th>\n",
       "      <td>Jilin</td>\n",
       "      <td>12171</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H5</td>\n",
       "      <td>4000000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1976 12:00:00 AM</td>\n",
       "      <td>44.05000</td>\n",
       "      <td>126.16667</td>\n",
       "      <td>(44.05, 126.16667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>506</th>\n",
       "      <td>Kunya-Urgench</td>\n",
       "      <td>12379</td>\n",
       "      <td>Valid</td>\n",
       "      <td>H5</td>\n",
       "      <td>1100000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1998 12:00:00 AM</td>\n",
       "      <td>42.25000</td>\n",
       "      <td>59.20000</td>\n",
       "      <td>(42.25, 59.2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>707</th>\n",
       "      <td>Norton County</td>\n",
       "      <td>17922</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Aubrite</td>\n",
       "      <td>1100000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1948 12:00:00 AM</td>\n",
       "      <td>39.68333</td>\n",
       "      <td>-99.86667</td>\n",
       "      <td>(39.68333, -99.86667)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>920</th>\n",
       "      <td>Sikhote-Alin</td>\n",
       "      <td>23593</td>\n",
       "      <td>Valid</td>\n",
       "      <td>Iron, IIAB</td>\n",
       "      <td>23000000.0</td>\n",
       "      <td>Fell</td>\n",
       "      <td>01/01/1947 12:00:00 AM</td>\n",
       "      <td>46.16000</td>\n",
       "      <td>134.65333</td>\n",
       "      <td>(46.16, 134.65333)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name     id nametype    recclass    mass (g)  fall  \\\n",
       "29         Allende   2278    Valid         CV3   2000000.0  Fell   \n",
       "419          Jilin  12171    Valid          H5   4000000.0  Fell   \n",
       "506  Kunya-Urgench  12379    Valid          H5   1100000.0  Fell   \n",
       "707  Norton County  17922    Valid     Aubrite   1100000.0  Fell   \n",
       "920   Sikhote-Alin  23593    Valid  Iron, IIAB  23000000.0  Fell   \n",
       "\n",
       "                       year    reclat    reclong             GeoLocation  \n",
       "29   01/01/1969 12:00:00 AM  26.96667 -105.31667  (26.96667, -105.31667)  \n",
       "419  01/01/1976 12:00:00 AM  44.05000  126.16667      (44.05, 126.16667)  \n",
       "506  01/01/1998 12:00:00 AM  42.25000   59.20000           (42.25, 59.2)  \n",
       "707  01/01/1948 12:00:00 AM  39.68333  -99.86667   (39.68333, -99.86667)  \n",
       "920  01/01/1947 12:00:00 AM  46.16000  134.65333      (46.16, 134.65333)  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.query(\"`mass (g)` > 1e6 and fall == 'Fell'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a8b60a0-7f33-411f-a11e-34d8d6dfe670",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "*Tip: Here, we can use both logical operators and bitwise operators.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b59a3c2-5515-43b1-b23e-6a6c747b8218",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Learning Path\n",
    "\n",
    "1. Anatomy of a DataFrame\n",
    "2. Creating DataFrames\n",
    "3. Inspecting the data\n",
    "4. Extracting subsets\n",
    "5. **Calculating summary statistics**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96a7b264-7cfc-403f-8c69-380e20ae163f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_calculating-summary-statistics"
    ]
   },
   "source": [
    "## Calculating summary statistics\n",
    "\n",
    "In the next section of this workshop, we will discuss data cleaning for a more meaningful analysis of our datasets; however, we can already extract some interesting insights from the `meteorites` data by calculating summary statistics."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bb77a5b-a57f-41f4-8131-4704f47a3b42",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_value-counts"
    ]
   },
   "source": [
    "#### How many of the meteorites were found versus observed falling?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "78de2225-84df-4df2-8388-30a6b2e22877",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fall\n",
       "Found    44609\n",
       "Fell      1107\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.fall.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba5f43b1-9070-46a8-bc8e-ec4502a8e1c2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "[The Meteoritical Society states](https://www.lpi.usra.edu/meteor/metbull.php?code=56147) that \"relict meteorites are composed mostly of terrestrial minerals, but are thought to have once been meteorites.\" What proportion of the data are relict meteorites? Let's verify these were all found versus observed falling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d53bb60a-aaaa-41fc-9779-c74ffab634bc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nametype  fall \n",
       "Valid     Found    0.974145\n",
       "          Fell     0.024215\n",
       "Relict    Found    0.001641\n",
       "Name: proportion, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.value_counts(subset=['nametype', 'fall'], normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fdb3a88-1c1f-4a37-84ed-d521be8fd8b0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_mean"
    ]
   },
   "source": [
    "#### What was the mass of the average meterorite?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1770a856-cb3d-42a4-abef-656198b9cdab",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(13278.078548601512)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites['mass (g)'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91b223c6-e174-4b8c-89ec-dc4538f36873",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "**Important**: The mean isn't always the best measure of central tendency. If there are outliers in the distribution, the mean will be skewed. Here, the mean is being pulled higher by some very heavy meteorites &ndash; the distribution is [right-skewed](https://www.analyticsvidhya.com/blog/2020/07/what-is-skewness-statistics/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7da8f8d-2838-4f3a-a011-1623c16b5c30",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_quantiles"
    ]
   },
   "source": [
    "Taking a look at some quantiles at the extremes of the distribution shows that the mean is between the 95th and 99th percentile of the distribution, so it isn't a good measure of central tendency here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0821d8e4-6096-4506-8d84-2f0f0ed09bd7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.01        0.44\n",
       "0.05        1.10\n",
       "0.50       32.60\n",
       "0.95     4000.00\n",
       "0.99    50600.00\n",
       "Name: mass (g), dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites['mass (g)'].quantile([0.01, 0.05, 0.5, 0.95, 0.99])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba56ba7-cb62-46a1-81b6-5117a2210460",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "A better measure in this case is the median (50th percentile), since it is robust to outliers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4aca7862-5a00-4f9e-88d8-004c6d680e0d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(32.6)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites['mass (g)'].median()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "377ade97-183d-428b-9b56-2cc1a1d3915f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_max-and-idxmax"
    ]
   },
   "source": [
    "#### What was the mass of the heaviest meteorite?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "35cd0108-7ef5-4911-bc6d-56249dff6b42",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(60000000.0)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites['mass (g)'].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6efc5b1b-413b-4378-9656-7c4d6001dc27",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Let's extract the information on this meteorite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "edf8ab72-ba40-4eb9-9015-37c0d60ef86d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name                             Hoba\n",
       "id                              11890\n",
       "nametype                        Valid\n",
       "recclass                    Iron, IVB\n",
       "mass (g)                   60000000.0\n",
       "fall                            Found\n",
       "year           01/01/1920 12:00:00 AM\n",
       "reclat                      -19.58333\n",
       "reclong                      17.91667\n",
       "GeoLocation     (-19.58333, 17.91667)\n",
       "Name: 16392, dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.loc[meteorites['mass (g)'].idxmax()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65367403-e010-4dc7-8cef-d280a8c94fd3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "*Fun fact: This meteorite landed in Namibia and is a tourist attraction.*\n",
    "\n",
    "<div>\n",
    "    <img width=\"25%\" style=\"margin-top: -5px; margin-bottom: -5px; min-width: 200px\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/a/a7/Hoba_meteorite_%2815682150765%29.jpg/1280px-Hoba_meteorite_%2815682150765%29.jpg\" alt=\"Hoba meteorite (source: Wikipedia)\"/>\n",
    "    <div><small><em>Source: Wikipedia</em></small></div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d14207-0f94-4204-927e-444535c7dd2d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_unique-and-nunique"
    ]
   },
   "source": [
    "#### How many different types of meteorite classes are represented in this dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "0301e74e-bdfd-4b9f-bf26-edd0c3ad5f39",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "466"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.recclass.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e55fe68e-5bf4-4b5e-b2c7-6a896743a5f6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5ad8d82a-aeaf-4d8b-bf5f-22b2ba06fad3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['L5', 'H6', 'EH4', 'Acapulcoite', 'L6', 'LL3-6', 'H5', 'L',\n",
       "       'Diogenite-pm', 'Unknown', 'H4', 'H', 'Iron, IVA', 'CR2-an'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.recclass.unique()[:14]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dc4f077-48b2-49ca-a902-faac7066f479",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "*Note: All fields preceded with \"rec\" are the values recommended by The Meteoritical Society. Check out [this Wikipedia article](https://en.wikipedia.org/wiki/Meteorite_classification) for some information on meteorite classes.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b506e37-2929-42be-a82d-eb31491bf352",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_describe-method"
    ]
   },
   "source": [
    "#### Get some summary statistics on the data itself\n",
    "We can get common summary statistics for all columns at once. By default, this will only be numeric columns, but here, we will summarize everything together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7bbf2092-31bd-4473-a98c-cfe243f9e7b1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>id</th>\n",
       "      <th>nametype</th>\n",
       "      <th>recclass</th>\n",
       "      <th>mass (g)</th>\n",
       "      <th>fall</th>\n",
       "      <th>year</th>\n",
       "      <th>reclat</th>\n",
       "      <th>reclong</th>\n",
       "      <th>GeoLocation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>45716</td>\n",
       "      <td>45716.000000</td>\n",
       "      <td>45716</td>\n",
       "      <td>45716</td>\n",
       "      <td>4.558500e+04</td>\n",
       "      <td>45716</td>\n",
       "      <td>45425</td>\n",
       "      <td>38401.000000</td>\n",
       "      <td>38401.000000</td>\n",
       "      <td>38401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>45716</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>466</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>266</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Aachen</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Valid</td>\n",
       "      <td>L6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Found</td>\n",
       "      <td>01/01/2003 12:00:00 AM</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(0.0, 0.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>45641</td>\n",
       "      <td>8285</td>\n",
       "      <td>NaN</td>\n",
       "      <td>44609</td>\n",
       "      <td>3323</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>26889.735104</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.327808e+04</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-39.122580</td>\n",
       "      <td>61.074319</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>16860.683030</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.749889e+05</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.378511</td>\n",
       "      <td>80.647298</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-87.366670</td>\n",
       "      <td>-165.433330</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>12688.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.200000e+00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-76.714240</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>24261.500000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.260000e+01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-71.500000</td>\n",
       "      <td>35.666670</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>40656.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.026000e+02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>157.166670</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>57458.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.000000e+07</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>81.166670</td>\n",
       "      <td>354.473330</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          name            id nametype recclass      mass (g)   fall  \\\n",
       "count    45716  45716.000000    45716    45716  4.558500e+04  45716   \n",
       "unique   45716           NaN        2      466           NaN      2   \n",
       "top     Aachen           NaN    Valid       L6           NaN  Found   \n",
       "freq         1           NaN    45641     8285           NaN  44609   \n",
       "mean       NaN  26889.735104      NaN      NaN  1.327808e+04    NaN   \n",
       "std        NaN  16860.683030      NaN      NaN  5.749889e+05    NaN   \n",
       "min        NaN      1.000000      NaN      NaN  0.000000e+00    NaN   \n",
       "25%        NaN  12688.750000      NaN      NaN  7.200000e+00    NaN   \n",
       "50%        NaN  24261.500000      NaN      NaN  3.260000e+01    NaN   \n",
       "75%        NaN  40656.750000      NaN      NaN  2.026000e+02    NaN   \n",
       "max        NaN  57458.000000      NaN      NaN  6.000000e+07    NaN   \n",
       "\n",
       "                          year        reclat       reclong GeoLocation  \n",
       "count                    45425  38401.000000  38401.000000       38401  \n",
       "unique                     266           NaN           NaN       17100  \n",
       "top     01/01/2003 12:00:00 AM           NaN           NaN  (0.0, 0.0)  \n",
       "freq                      3323           NaN           NaN        6214  \n",
       "mean                       NaN    -39.122580     61.074319         NaN  \n",
       "std                        NaN     46.378511     80.647298         NaN  \n",
       "min                        NaN    -87.366670   -165.433330         NaN  \n",
       "25%                        NaN    -76.714240      0.000000         NaN  \n",
       "50%                        NaN    -71.500000     35.666670         NaN  \n",
       "75%                        NaN      0.000000    157.166670         NaN  \n",
       "max                        NaN     81.166670    354.473330         NaN  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meteorites.describe(include='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bc16e99-4168-43e2-b770-e1939712c744",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "**Important**: `NaN` values signify missing data. For instance, the `fall` column contains strings, so there is no value for `mean`; likewise, `mass (g)` is numeric, so we don't have entries for the categorical summary statistics (`unique`, `top`, `freq`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23eb0e09-a60c-4228-a85f-cfbd7c414666",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Check out the documentation for more descriptive statistics:\n",
    "\n",
    "- [Series](https://pandas.pydata.org/docs/reference/series.html#computations-descriptive-stats)\n",
    "- [DataFrame](https://pandas.pydata.org/docs/reference/frame.html#computations-descriptive-stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d494f19a-35d2-45ce-8042-f411d61d616d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "id_exercises-section-1-part-2"
    ]
   },
   "source": [
    "### Exercise 1.3\n",
    "\n",
    "##### Using the data in the `2019_Yellow_Taxi_Trip_Data.csv` file, calculate summary statistics for the `fare_amount`, `tip_amount`, `tolls_amount`, and `total_amount` columns.\n",
    "\n",
    "### Exercise 1.4\n",
    "\n",
    "##### Isolate the `fare_amount`, `tip_amount`, `tolls_amount`, and `total_amount` for the longest trip by distance (`trip_distance`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfc7ca90-3bb4-4dd6-97fd-8afe50b5dfd3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_solutions-section-1-part-2"
    ]
   },
   "source": [
    "### Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "769a4fc4-8648-4929-a736-d862370bc94e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_solution-1-3"
    ]
   },
   "source": [
    "##### Exercise 1.3 &ndash; Using the data in the `2019_Yellow_Taxi_Trip_Data.csv` file, calculate summary statistics for the `fare_amount`, `tip_amount`, `tolls_amount`, and `total_amount` columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0f348a93-32c6-474c-acf9-c4fd1a80aa26",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>total_amount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>15.106313</td>\n",
       "      <td>2.634494</td>\n",
       "      <td>0.623447</td>\n",
       "      <td>22.564659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>13.954762</td>\n",
       "      <td>3.409800</td>\n",
       "      <td>6.437507</td>\n",
       "      <td>19.209255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-52.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-6.120000</td>\n",
       "      <td>-65.920000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>12.375000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>16.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>16.000000</td>\n",
       "      <td>3.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>22.880000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>176.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>612.000000</td>\n",
       "      <td>671.800000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        fare_amount    tip_amount  tolls_amount  total_amount\n",
       "count  10000.000000  10000.000000  10000.000000  10000.000000\n",
       "mean      15.106313      2.634494      0.623447     22.564659\n",
       "std       13.954762      3.409800      6.437507     19.209255\n",
       "min      -52.000000      0.000000     -6.120000    -65.920000\n",
       "25%        7.000000      0.000000      0.000000     12.375000\n",
       "50%       10.000000      2.000000      0.000000     16.300000\n",
       "75%       16.000000      3.250000      0.000000     22.880000\n",
       "max      176.000000     43.000000    612.000000    671.800000"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "taxis = pd.read_csv('../data/2019_Yellow_Taxi_Trip_Data.csv')\n",
    "taxis[['fare_amount', 'tip_amount', 'tolls_amount', 'total_amount']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99d54120-baba-4173-ba5e-78ce4e6bff08",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "id_solution-1-4"
    ]
   },
   "source": [
    "##### Exercise 1.4 &ndash; Isolate the `fare_amount`, `tip_amount`, `tolls_amount`, and `total_amount` for the longest trip by distance (`trip_distance`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "14059a98-6ddf-417d-98fa-cd5f36130e03",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fare_amount      176.0\n",
       "tip_amount       18.29\n",
       "tolls_amount      6.12\n",
       "total_amount    201.21\n",
       "Name: 8338, dtype: object"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxis.loc[\n",
    "    taxis.trip_distance.idxmax(), \n",
    "    ['fare_amount', 'tip_amount', 'tolls_amount', 'total_amount']\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ebc5fdc-39fb-40c1-a1f1-e89d2186b826",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Section 1 Complete 🎉\n",
    "\n",
    "<img width=\"30%\" src=\"https://www.pngkey.com/png/detail/11-115894_giant-panda-bear-cuteness-panda-with-bamboo-cartoon.png\" alt=\"Giant Panda Bear Cuteness - Panda With Bamboo Cartoon@pngkey.com\" style=\"min-width: 300px\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}