1_1_data_scraper.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/LouisVanLangendonck/UPC-AML-ArchitectureClassif/blob/main/data_scraper.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Scraper for Architectural Stlye Images"
      ],
      "metadata": {
        "id": "4iTEDxp8J7by"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This image web scraper is written in function of using it in Google Colab. Use outside of this context will need small adaptations to the code. \n",
        "\n",
        "Start by defining where the Architecture Classification MAIN DIRECTORY is located (the data directory substructure will automatically be created, and images downloaded in accordingly)"
      ],
      "metadata": {
        "id": "mIrekUQrIUD0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "project_dir = '/content/drive/MyDrive/AML'"
      ],
      "metadata": {
        "id": "lwZPpZbdGHe3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets create the directory substructure..."
      ],
      "metadata": {
        "id": "R-5hTjIlI13X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive', force_remount = True)\n",
        "\n",
        "try:\n",
        "  os.makedirs(project_dir+'/data/unzipped/all_data/')\n",
        "except:\n",
        "  pass\n",
        "\n",
        "data_dir = project_dir + '/data/unzipped/all_data/'\n",
        "style_list = ['baroque', 'contemporary', 'gothic', 'modernism', 'neoclassicism', 'noucentisme', 'renaissance', 'romanesque']\n",
        "\n",
        "for style in style_list:\n",
        "  try:\n",
        "    os.mkdir(os.path.join(data_dir, style))\n",
        "  except:\n",
        "    pass"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q_tNoI_g4AcA",
        "outputId": "044012ba-fb70-445b-ca33-e628c68d9bac"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Necessary Webdriver installs and package parametrisation in order to run Selenium on Google Colab.. "
      ],
      "metadata": {
        "id": "SxwRYBw0I57v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!apt-get update # to update ubuntu to correctly run apt install\n",
        "!apt install chromium-chromedriver\n",
        "!cp /usr/lib/chromium-browser/chromedriver /usr/bin\n",
        "import sys\n",
        "sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')\n",
        "from selenium import webdriver\n",
        "chrome_options = webdriver.ChromeOptions()\n",
        "chrome_options.add_argument('--headless')\n",
        "chrome_options.add_argument('--no-sandbox')\n",
        "chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "wd = webdriver.Chrome('chromedriver',options=chrome_options)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BEjBfxwwHQT8",
        "outputId": "bb12419e-a7f8-4d7d-bab5-87d3b99749e4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\r0% [Working]\r            \rHit:1 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease\n",
            "\r0% [Connecting to archive.ubuntu.com (91.189.91.39)] [Connecting to security.ub\r0% [1 InRelease gpgv 3,626 B] [Waiting for headers] [Connecting to security.ubu\r                                                                               \rIgn:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "\r0% [1 InRelease gpgv 3,626 B] [Waiting for headers] [Connecting to security.ubu\r                                                                               \rHit:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "\r0% [1 InRelease gpgv 3,626 B] [Waiting for headers] [Waiting for headers] [Conn\r                                                                               \rHit:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
            "\r0% [1 InRelease gpgv 3,626 B] [Waiting for headers] [Waiting for headers] [Wait\r                                                                               \rHit:5 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "Hit:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease\n",
            "Hit:7 http://security.ubuntu.com/ubuntu bionic-security InRelease\n",
            "Hit:8 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease\n",
            "Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease\n",
            "Hit:10 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n",
            "Hit:12 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease\n",
            "Hit:13 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
            "Reading package lists... Done\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "chromium-chromedriver is already the newest version (108.0.5359.71-0ubuntu0.18.04.5).\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-460\n",
            "Use 'apt autoremove' to remove it.\n",
            "0 upgraded, 0 newly installed, 0 to remove and 20 not upgraded.\n",
            "cp: '/usr/lib/chromium-browser/chromedriver' and '/usr/bin/chromedriver' are the same file\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The actual functions that manouver through, download and correctly place the different images of all buildings of a respective style to be found on 'http://invarquit.cultura.gencat.cat/Cerca/'. "
      ],
      "metadata": {
        "id": "SNUFINzFJBbP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from selenium import webdriver\n",
        "from selenium.webdriver.common.by import By\n",
        "from selenium.webdriver.support.ui import Select\n",
        "import time \n",
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "import urllib.request\n",
        "\n",
        "driver = wd\n",
        "\n",
        "def scrape_style(url, style_name):\n",
        "    big_iter = 0\n",
        "    base = 'http://invarquit.cultura.gencat.cat/Cerca/'  \n",
        "    driver.get(url)\n",
        "    first_building = driver.find_elements(By.CLASS_NAME, \"enlacedestacado\")[0] #Do this only once. From here on uses other button\n",
        "    first_building.click()\n",
        "    download_images(driver.current_url, style=style_name, big_it=big_iter) #To click trough all images and download each for first \n",
        "    next_buttons = driver.find_elements(By.TAG_NAME, 'img') \n",
        "    for button in next_buttons:\n",
        "        if 'endavant' in button.get_attribute('src'):\n",
        "            if \"public\" in button.get_attribute('title'):\n",
        "                next_building = button\n",
        "    next_building.click() #Go to next building\n",
        "    contin=True\n",
        "    while contin:\n",
        "        big_iter += 1\n",
        "        next_buttons = driver.find_elements(By.TAG_NAME, 'img')\n",
        "        for button in next_buttons:\n",
        "            if 'endavant' in button.get_attribute('src'):\n",
        "                if \"public\" in button.get_attribute('title') and not \"Off\" in button.get_attribute('title'):\n",
        "                    next_building = button #There is a next building \n",
        "                elif \"public\" in button.get_attribute('title') and \"Off\" in button.get_attribute('title'): #Last building of style\n",
        "                    download_images(driver.current_url, style=style_name, big_it=big_iter) #To click trough all images and download each \n",
        "                    contin = False   \n",
        "        if contin:\n",
        "            next_building.click()\n",
        "            download_images(driver.current_url, style=style_name, big_it=big_iter) #To click trough all images and download each \n",
        "\n",
        "\n",
        "def download_images(url, style, big_it):\n",
        "    it = 0\n",
        "    try:\n",
        "        page = requests.get(url)\n",
        "        soup = BeautifulSoup(page.content, \"html.parser\")\n",
        "        candid_dupl = soup.find_all(class_ = \"taulesDeDins\") \n",
        "        if len(candid_dupl[0]) == 5:                    #To check if only a single style label is assigned to the building\n",
        "            img_source = driver.find_element(By.ID, \"imatgeACanviar\").get_attribute(\"src\")\n",
        "            urllib.request.urlretrieve(img_source, '{}/{}/{}-{}-{}.jpg'.format(data_dir, style,style, big_it, it)) #Naming convention\n",
        "            next_buttons = driver.find_elements(By.TAG_NAME, 'img')\n",
        "            for button in next_buttons:\n",
        "                if 'endavant' in button.get_attribute('src'):\n",
        "                    if not \"public\" in button.get_attribute('title'):\n",
        "                        next_img = button\n",
        "            next_img.click()\n",
        "            new_img_source = driver.find_element(By.ID, \"imatgeACanviar\").get_attribute(\"src\")\n",
        "            while new_img_source != img_source and it < 7: #Download at most 7 images\n",
        "                it += 1\n",
        "                urllib.request.urlretrieve(new_img_source, '{}/{}/{}-{}-{}.jpg'.format(data_dir,style,style, big_it, it))\n",
        "                next_buttons = driver.find_elements(By.TAG_NAME, 'img')\n",
        "                for button in next_buttons:\n",
        "                    if 'endavant' in button.get_attribute('src'):\n",
        "                        if not \"public\" in button.get_attribute('title'):\n",
        "                            next_img = button\n",
        "                next_img.click()\n",
        "                img_source = new_img_source\n",
        "                new_img_source = driver.find_element(By.ID, \"imatgeACanviar\").get_attribute(\"src\")   \n",
        "    except:\n",
        "        pass"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tOpCr2Rp4YIH",
        "outputId": "8f631790-38c8-45b6-ccff-5031d9a3d300"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: selenium in /usr/local/lib/python3.8/dist-packages (4.7.2)\n",
            "Requirement already satisfied: urllib3[socks]~=1.26 in /usr/local/lib/python3.8/dist-packages (from selenium) (1.26.13)\n",
            "Requirement already satisfied: trio~=0.17 in /usr/local/lib/python3.8/dist-packages (from selenium) (0.22.0)\n",
            "Requirement already satisfied: trio-websocket~=0.9 in /usr/local/lib/python3.8/dist-packages (from selenium) (0.9.2)\n",
            "Requirement already satisfied: certifi>=2021.10.8 in /usr/local/lib/python3.8/dist-packages (from selenium) (2022.12.7)\n",
            "Requirement already satisfied: exceptiongroup>=1.0.0rc9 in /usr/local/lib/python3.8/dist-packages (from trio~=0.17->selenium) (1.0.4)\n",
            "Requirement already satisfied: async-generator>=1.9 in /usr/local/lib/python3.8/dist-packages (from trio~=0.17->selenium) (1.10)\n",
            "Requirement already satisfied: sortedcontainers in /usr/local/lib/python3.8/dist-packages (from trio~=0.17->selenium) (2.4.0)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.8/dist-packages (from trio~=0.17->selenium) (2.10)\n",
            "Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.8/dist-packages (from trio~=0.17->selenium) (22.1.0)\n",
            "Requirement already satisfied: outcome in /usr/local/lib/python3.8/dist-packages (from trio~=0.17->selenium) (1.2.0)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.8/dist-packages (from trio~=0.17->selenium) (1.3.0)\n",
            "Requirement already satisfied: wsproto>=0.14 in /usr/local/lib/python3.8/dist-packages (from trio-websocket~=0.9->selenium) (1.2.0)\n",
            "Requirement already satisfied: PySocks!=1.5.7,<2.0,>=1.5.6 in /usr/local/lib/python3.8/dist-packages (from urllib3[socks]~=1.26->selenium) (1.7.1)\n",
            "Requirement already satisfied: h11<1,>=0.9.0 in /usr/local/lib/python3.8/dist-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium) (0.14.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here the function is iteratively called for the styles specified. Note that it takes a very long time to run the whole loop... Check the files in the data file to see if the images are actually being loaded in! "
      ],
      "metadata": {
        "id": "3iLQHm3OJaPb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "style_list = ['baroque', 'contemporary', 'gothic', 'modernism', 'neoclassicism', 'noucentisme', 'renaissance', 'romanesque']\n",
        "corr_url = ['http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K0JhcnJvYyU%3D', 'http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K0RhcnJlcmVzIHRlbmTDqG5jaWVzJQ%3D%3D',\n",
        "'http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K0fDsnRpYyU%3D', 'http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K01vZGVybmlzbWUl', 'http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K05lb2NsYXNzaWNpc21lJQ%3D%3D',\n",
        "'http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K05vdWNlbnRpc21lJQ%3D%3D', 'http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K1JlbmFpeGVtZW50JQ%3D%3D', \n",
        "'http://invarquit.cultura.gencat.cat/Cerca/Llista?Consulta=MCU4K1JvbcOgbmljJQ%3D%3D'] #Loop of all links necessary. Can easily be extended.\n",
        "\n",
        "for idx, style in enumerate(style_list): #Actual command to run and scrape all. Note that everything will take a long time. \n",
        "    URL = corr_url[idx]\n",
        "    scrape_style(URL, style)"
      ],
      "metadata": {
        "id": "JsXAIusn4drz"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}