Scraping-Github-Top-Repositories_doc-Py.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1638e1ed",
   "metadata": {},
   "source": [
    "# Scraping Top Repositories Topics of Github\n",
    "\n",
    "**MISSIONS**\n",
    "\n",
    "- Introduction to `web scraping`.\n",
    "- Intorduction to `github` and the problem statement\n",
    "- Mention the tools and libraries that're being used(Python, requests, beautiful Soup, Pandas)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41cd8298",
   "metadata": {},
   "source": [
    "1. Know more about requests: https://requests.readthedocs.io/en/latest/\n",
    "2. Know more about beautifulSoup: https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
    "3. Know more about Pandas: https://pandas.pydata.org/docs/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9712d59",
   "metadata": {},
   "source": [
    "**OUTLINES**\n",
    "\n",
    "Steps that are followed:\n",
    "\n",
    "* Going to scrape https://github.com/topics.\n",
    "* Get a list of topics. For each topic, will get topic title, topic page URL and topic description.\n",
    "* For each topic, will get top 20 repositories in the topic from the topic page\n",
    "* For each repository, will get the repo name, username, stars and repo URL\n",
    "* For each topic, will create a CSV file in the following format:\n",
    "\n",
    "\n",
    "Repo Name,Username,Stars,Repo URL\n",
    "three.js,mardoob,92200,https://github.com/mrdoob/three.js\n",
    "react-three-fibre,pmndrs,22700,https://github.com/pmndrs/react-three-fiber\n",
    "libgdx,libgdx,21500,https://github.com/libgdx/libgdx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7a31d7a",
   "metadata": {},
   "source": [
    "## Scrape the list of topics from Github\n",
    "\n",
    "* Use `requests` library to download the page\n",
    "* Use `bs4` to parse and extract information\n",
    "* Convert to a Pandas dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c55ed63f",
   "metadata": {},
   "source": [
    "**Write a function to download the page**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "8160f48c",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import os\n",
    "def get_topics_page():\n",
    "    topics_url = 'https://github.com/topics'\n",
    "    response = requests.get(topics_url)\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "        \n",
    "    doc = BeautifulSoup(response.text, 'html.parser')\n",
    "    return doc\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "82f83604",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = get_topics_page()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "edff11c8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.BeautifulSoup"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71d92b02",
   "metadata": {},
   "source": [
    "**Create some helper functions to parse information from the page**\n",
    "\n",
    "To get topic titles, we pick `p` tags with the class\n",
    "\n",
    "![](https://i.imgur.com/PYd4XVw.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "fb773f04",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_titles(doc):\n",
    "    topic_title_p_tags = doc.find_all('p', {'class': 'f3 lh-condensed mb-0 mt-1 Link--primary'})\n",
    "    topic_titles = []\n",
    "    for tag in topic_title_p_tags:\n",
    "        topic_titles.append(tag.text)\n",
    "    \n",
    "    return topic_titles\n",
    "    r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ed21291",
   "metadata": {},
   "source": [
    "`get_topic_titles` is used to get the list of titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "09eb0a56",
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = get_topic_titles(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "ec40ce68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "6ec71fb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3D', 'Ajax', 'Algorithm']"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c10bbc9d",
   "metadata": {},
   "source": [
    "**Similarly, we've defined functions for descriptions and URLs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "fb58ad57",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "    \n",
    "def get_topic_descs(doc):\n",
    "    topic_desc_tags = doc.find_all('p', {'class': 'f5 color-fg-muted mb-0 mt-1'})\n",
    "    topic_descs = []\n",
    "    for tag in topic_desc_tags:\n",
    "        topic_descs.append(tag.text.strip())\n",
    "    return topic_descs\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d875063d",
   "metadata": {},
   "source": [
    "`get_topic_descs` is used to get the list of description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "ba3ed943",
   "metadata": {},
   "outputs": [],
   "source": [
    "descs = get_topic_descs(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "89df1cbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(descs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "464cce42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3D refers to the use of three-dimensional graphics, modeling, and animation in various industries.',\n",
       " 'Ajax is a technique for creating interactive web applications.']"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "descs[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7522ec42",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65134509",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "ec2ec58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_urls(doc):\n",
    "    topic_link_tags = doc.find_all('a', {'class': 'no-underline flex-grow-0'})\n",
    "    topic_urls = []\n",
    "    base_url = 'https://github.com'\n",
    "\n",
    "    for tag in topic_link_tags:\n",
    "        topic_urls.append(base_url + tag['href'])\n",
    "\n",
    "    return topic_urls\n",
    "    \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a848e99",
   "metadata": {},
   "source": [
    "`get_topic_urls` is used to get the list of the urls of the topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "115f5b80",
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = get_topic_urls(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "d4700829",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(urls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "e46737ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://github.com/topics/3d',\n",
       " 'https://github.com/topics/ajax',\n",
       " 'https://github.com/topics/algorithm',\n",
       " 'https://github.com/topics/amphp']"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urls[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c09d877",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1144c6aa",
   "metadata": {},
   "source": [
    "**Let's put this all together into a single function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "64acbda6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topics():\n",
    "    topics_url = 'https://github.com/topics'\n",
    "    response = requests.get(topics_url)\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "        \n",
    "    doc = BeautifulSoup(response.text, 'html.parser')\n",
    "    topics_dict = {\n",
    "        'title': get_topic_titles(doc),\n",
    "        'description': get_topic_descs(doc),\n",
    "        'url': get_topic_urls(doc)\n",
    "    \n",
    "    }\n",
    "        \n",
    "    return pd.DataFrame(topics_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f37e3f70",
   "metadata": {},
   "source": [
    "## Get the top 20 repositories from a topic page\n",
    "\n",
    "**MISSION:**\n",
    "\n",
    "- Getting individual top repositories for individual topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "7baad803",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_page(topic_url):\n",
    "       # Download the page\n",
    "    response = requests.get(topic_url)\n",
    "\n",
    "    # Check the successfull response\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "    # Parse using BeautifulSoup\n",
    "    topic_doc = BeautifulSoup(response.text, 'html.parser')\n",
    "\n",
    "    return topic_doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "1c37c6c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = get_topic_page('https://github.com/topics/3d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4e5edeb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "b3e5165b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_star_count(stars_str):\n",
    "    stars_str = stars_str.strip()\n",
    "    if stars_str[-1] == 'k':\n",
    "        return int(float(stars_str[:-1]) * 1000)\n",
    "    return int(stars_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "917e4203",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_url = 'https://github.com/topics'\n",
    "def get_repo_info(h3_tag, star_tag):\n",
    "    # returns all necessary info about a repository\n",
    "    a_tags = h3_tag.find_all('a')\n",
    "    username = a_tags[0].text.strip()\n",
    "    repo_name = a_tags[1].text.strip()\n",
    "    repo_url = base_url + a_tags[0]['href']\n",
    "    stars = parse_star_count(star_tag.text.strip())\n",
    "    return username, repo_name, stars, repo_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddbd1457",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "d5931cd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_repos(topic_doc):\n",
    "    # Get the h3 tags containing username, repo title and repo URL\n",
    "    h3_selection_class = 'f3 color-fg-muted text-normal lh-condensed'\n",
    "    repo_tags = topic_doc.find_all('h3', {'class': h3_selection_class})\n",
    "    # Get star tags\n",
    "    star_tags = topic_doc.find_all('span', {'class': 'Counter js-social-count'})\n",
    "    \n",
    "    # Get repo info\n",
    "    topic_repos_dict = {\n",
    "        'username': [], \n",
    "         'repo_name': [], \n",
    "         'stars': [],\n",
    "         'repo_url': []\n",
    "     }\n",
    "    for i in range(len(repo_tags)):\n",
    "        repo_info = get_repo_info(repo_tags[i], star_tags[i])\n",
    "        topic_repos_dict['username'].append(repo_info[0])\n",
    "        topic_repos_dict['repo_name'].append(repo_info[1])\n",
    "        topic_repos_dict['stars'].append(repo_info[2])\n",
    "        topic_repos_dict['repo_url'].append(repo_info[3])\n",
    "\n",
    "    return pd.DataFrame(topic_repos_dict)"
   ]
  },
  {
   "cell_type": "raw",
   "id": "97888aed",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "6df4ec3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topic(topic_url, path):\n",
    "    if os.path.exists(path):\n",
    "        print(\"The file {} already exists. Skipping...\".format(path))\n",
    "    topic_df = get_topic_repos(get_topic_page(topic_url))\n",
    "    topic_df.to_csv(path, index = None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13cf105c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "589dd885",
   "metadata": {},
   "source": [
    "# Putting it all together\n",
    "\n",
    "- We use a function to get the list of all topics\n",
    "- We use a function to create a CSV file for scraped repos from a topics page\n",
    "- Let's create a function to put them together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "c4fb3eef",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topics_repos():\n",
    "    print('Scraping list of topics')\n",
    "    topics_df = scrape_topics()\n",
    "    \n",
    "    os.makedirs('data', exist_ok=True)\n",
    "    \n",
    "    for index, row in topics_df.iterrows():\n",
    "        print('Scrapping top repositories for \"{}\"'.format(row['title']))\n",
    "        scrape_topic(row['url'], 'data/{}.csv'.format(row['title']))\n",
    "       "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f38f19c",
   "metadata": {},
   "source": [
    "Let's run  it to scrape the top repos for all the topics on the first page of https://github.com/topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "006f476e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping list of topics\n",
      "Scrapping top repositories for \"3D\"\n",
      "The file data/3D.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Ajax\"\n",
      "The file data/Ajax.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Algorithm\"\n",
      "The file data/Algorithm.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Amp\"\n",
      "The file data/Amp.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Android\"\n",
      "The file data/Android.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Angular\"\n",
      "The file data/Angular.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Ansible\"\n",
      "The file data/Ansible.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"API\"\n",
      "The file data/API.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Arduino\"\n",
      "The file data/Arduino.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"ASP.NET\"\n",
      "The file data/ASP.NET.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Atom\"\n",
      "The file data/Atom.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Awesome Lists\"\n",
      "The file data/Awesome Lists.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Amazon Web Services\"\n",
      "The file data/Amazon Web Services.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Azure\"\n",
      "The file data/Azure.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Babel\"\n",
      "The file data/Babel.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Bash\"\n",
      "The file data/Bash.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Bitcoin\"\n",
      "The file data/Bitcoin.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Bootstrap\"\n",
      "The file data/Bootstrap.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Bot\"\n",
      "The file data/Bot.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"C\"\n",
      "The file data/C.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Chrome\"\n",
      "The file data/Chrome.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Chrome extension\"\n",
      "The file data/Chrome extension.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Command line interface\"\n",
      "The file data/Command line interface.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Clojure\"\n",
      "The file data/Clojure.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Code quality\"\n",
      "The file data/Code quality.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Code review\"\n",
      "The file data/Code review.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Compiler\"\n",
      "The file data/Compiler.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"Continuous integration\"\n",
      "The file data/Continuous integration.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"COVID-19\"\n",
      "The file data/COVID-19.csv already exists. Skipping...\n",
      "Scrapping top repositories for \"C++\"\n",
      "The file data/C++.csv already exists. Skipping...\n"
     ]
    }
   ],
   "source": [
    "scrape_topics_repos()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9ea1e3a",
   "metadata": {},
   "source": [
    "**We can check the CSVs were created properly**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2df9f6",
   "metadata": {},
   "source": [
    "# References and Future Work\n",
    "\n",
    "\n",
    "References to link found useful\n",
    "- https://requests.readthedocs.io/en/latest/\n",
    "-  https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
    "-  https://pandas.pydata.org/docs/\n",
    "- https://jovian.ai/aakashns/python-web...\n",
    "- https://jovian.ai/aakashns/python-web..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81d26c00",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}