04_library_catalogue.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JSON and Library Catalog Data\n",
    "\n",
    "How can we extract data from APIs (machine-readable online data sources)?\n",
    "\n",
    "In this lesson, we look at how we can use data from the web.\n",
    "We will use real-world data from\n",
    "[The National Library of Norway](https://www.nb.no/).\n",
    "The National Library has a [search API](https://api.nb.no/)\n",
    "which we will use.\n",
    "\n",
    "## JSON\n",
    "\n",
    "JSON is a machine-readable data format.\n",
    "Machine-readable data makes it easy to read and process the information with a computer.\n",
    "JSON data is usually tree structured, with multiple levels containing information.\n",
    "This is kind of like a directory tree containing files.\n",
    "\n",
    "## Fetching Data\n",
    "To fetch data from the web, we can use a library called requests that makes this task quite easy.\n",
    "Since we are are fetching data in the JSON format, we will also import a library to decode JSON data.\n",
    "Libraries are collections of code written by others that we can utilize instead of\n",
    "writing everything from scratch ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to specify the URL to the data we want to fetch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"https://api.nb.no/catalog/v1/items?digitalAccessibleOnly=true&size=3&filter=mediatype:bøker&q=Bing,Jon\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We include some parameters that specifies which cases we want to load:\n",
    "- `digitalAccessibleOnly` specifies that we only want documents that are available in fulltext\n",
    "- `size` is the number of items to fetch\n",
    "- `filter` narrows the search, in this case to books\n",
    "- `q` is the search query\n",
    "\n",
    "More parameters are listed in the [API documentation](https://api.nb.no/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's fetch the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = requests.get(URL).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step both fetches and decodes the json data in one line. We can also do this step-by-step, to see how the process works.\n",
    "If you don't want to get into the details at this point, you can skip ahead to the section \"Using the data\". \n",
    "The server response also contains metadata, but we want the content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(URL)\n",
    "content = response.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the first 100 characters from the raw data. We can see the same data if we open the URL in a web browser:\n",
    "https://api.nb.no/catalog/v1/items?digitalAccessibleOnly=true&size=3&filter=mediatype:bøker&q=Bing,Jon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(content[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the data, we must decode them. We must specify the character set, which is often UTF-8. Then we decode the json format into a Python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = content.decode(\"utf-8\")\n",
    "data = json.loads(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print the data, however this is a lot of text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead, we can print only the keys using `list()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = list(data)\n",
    "print(keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our search results only contain the first few items. We say that we are viewing a *page* of the results.\n",
    "The field `page` contains information about the current page.\n",
    "This is a dictionary, and we can print the information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page = data['page']\n",
    "print(page)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The field `size` contains the number of hits in the database. This is usually different from the number of items we requested.\n",
    "If the `size` is zero, we don't have any results and need to check the query in the URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "size = page['size']\n",
    "print(size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks good. Let's fetch the list of items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedded = data['_embedded']\n",
    "items = embedded['items']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can inspect each item. Let's loop over the items and get some of the information.\n",
    "The data contains various metadata about each item, such as the item ID.\n",
    "\n",
    "It's often useful to look at the data in a web browser to get an overview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in items:\n",
    "    print(\"item ID:\", item['id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each item has a `metadata` dictionary with the title etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in items:\n",
    "    metadata = item['metadata']\n",
    "    print(\"Item title:\", metadata['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:green\"> Exercise: Creator </span>\n",
    "\n",
    "Complete the code below to print the creator of each item.\n",
    "You will need to browse the [data](https://api.nb.no/catalog/v1/items?digitalAccessibleOnly=true&size=3&filter=mediatype:bøker&q=Bing,Jon)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "URL = \"https://api.nb.no/catalog/v1/items?digitalAccessibleOnly=true&size=3&filter=mediatype:bøker&q=Bing,Jon\"\n",
    "data = requests.get(URL).json()\n",
    "embedded = data['_embedded']\n",
    "items = embedded['items']\n",
    "\n",
    "for item in items:\n",
    "    metadata = item['metadata']\n",
    "    print(\"Item title:\", metadata['title'])\n",
    "    #your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Following the Path\n",
    "\n",
    "As mentioned, JSON data is a tree structure. It can contain many nested levels. In that case, we need to follow the path to find the entry we're looking for.\n",
    "It's usually advisable to follow the path one step at a time. This makes it easier to find errors in our programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"https://api.nb.no/catalog/v1/items?digitalAccessibleOnly=true&size=3&filter=mediatype:bøker&q=Bing,Jon\"\n",
    "data = requests.get(URL).json()\n",
    "embedded = data['_embedded']\n",
    "items = embedded['items']\n",
    "\n",
    "for item in items:\n",
    "    metadata = item['metadata']\n",
    "    print(\"Item title:\", metadata['title'])\n",
    "    \n",
    "    # Step-by-step:\n",
    "    links = item['_links']\n",
    "    self = links['self']\n",
    "    href = self['href']\n",
    "    print(\"Item URL:\", href)\n",
    "\n",
    "    # We start a new path from links:\n",
    "    thumbnail_large = links['thumbnail_large']\n",
    "    thumbnail_URL = thumbnail_large['href'] # have already used the name 'href'\n",
    "    print('Thumbnail:', thumbnail_URL)\n",
    "\n",
    "    # Extra linebreak:\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "###  <span style=\"color:green\"> Exercise: Presentation and URN </span>\n",
    "\n",
    "The field `presentation` contains a link to the full text.\n",
    "Complete the code below to print the `presentation` URL and the 'URN' of each item.\n",
    "\n",
    "You will need to browse the [data](https://api.nb.no/catalog/v1/items?digitalAccessibleOnly=true&size=3&filter=mediatype:bøker&q=Bing,Jon)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "URL = \"https://api.nb.no/catalog/v1/items?digitalAccessibleOnly=true&size=3&filter=mediatype:bøker&q=Bing,Jon\"\n",
    "data = requests.get(URL).json()\n",
    "embedded = data['_embedded']\n",
    "items = embedded['items']\n",
    "\n",
    "for item in items:\n",
    "    metadata = item['metadata']\n",
    "    print(\"Item title:\", metadata['title'])\n",
    "    #your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Lists\n",
    "\n",
    "Each item in the data set contains a list of one or more creators.\n",
    "These lists are located in each item's `metadata` field.\n",
    "We can use a `for`-loop to process the list items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in items:\n",
    "    metadata = item['metadata']\n",
    "    print(\"Item title:\", metadata['title'])\n",
    "    creators = metadata['creators']\n",
    "    for creator in creators:\n",
    "        print('Creator:',creator)\n",
    "    print() #insert empty line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  <span style=\"color:blue\">Key Points</span>\n",
    "\n",
    "- The `requests` library can be used to fetch data from the web.\n",
    "- Many data providers provide an API from which we can fetch data programatically.\n",
    "- Parameters can be used to control what data we get from an API.\n",
    "- Most APIs provide data in the JSON format, and JSON is well supported in Python.\n",
    "- Additional filtering and processing of the retrieved data can be done using loops and conditions (`if`-statements, next chapter)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}