README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``This file was built from an IPython Notebook. Download README.ipynb on Github to poke around.``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# wbpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Python interface to the World Bank Indicators and Climate APIs. \n",
    "\n",
    "* [Readthedocs](http://wbpy.readthedocs.org/en/latest)\n",
    "* [Github source](https://github.com/mattduck/wbpy)\n",
    "* [World Bank API docs](http://data.worldbank.org/developers)\n",
    "\n",
    "The Indicators API lets you access a large number of world development indicators - country data on education, environment, gender, health, population, poverty, technology, and more.\n",
    "\n",
    "The Climate API lets you access modelled and historical data for temperature and precipitation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why use wbpy?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Dataset models let you access processed data and associated metadata in different formats. \n",
    "* If you don't want processed data objects, you can still access the raw JSON response.\n",
    "* Single method calls to do the equivalent of multiple API requests, eg. wbpy handles the specific date pairs which would otherwise be required for the Climate API. \n",
    "* Works with both ISO 1366 alpha-2 and alpha-3 country codes (the web APIs mostly just support alpha-3). \n",
    "\n",
    "Elsewhere, there is also [wbdata](https://github.com/OliverSherouse/wbdata), a wrapper for the Indicators API which supports Pandas structures and has some command-line functionality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``pip install wbpy``, or download the source code and ``python setup.py install``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contributors\n",
    "\n",
    "- [@bcipolli](https://github.com/bcipolli) upgraded wbpy to support Python 3 and v2 of the world bank API.\n",
    "\n",
    "\n",
    "## Development and maintenance\n",
    "\n",
    "This project was unmaintained for a couple of years, although was updated in July 2020 to support Python 3 and to use the v2 endpoint of the API, as v1 has not been supported for a while (thanks [@bcipolli](https://github.com/bcipolli)!). Although I'm not actively adding new features or looking for issues, I'm happy to accept contributions, and to provide commit access if anybody wants to work on the project.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indicators API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a small case where we already know what API codes to use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/countries/GBR;FRA;JPN/indicators/SP.POP.TOTL?date=2010%3A2012&format=json&per_page=10000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<wbpy.indicators.IndicatorDataset('SP.POP.TOTL', 'Population, total') with id: 140421139962456>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import wbpy\n",
    "from pprint import pprint\n",
    "\n",
    "api = wbpy.IndicatorAPI()\n",
    "\n",
    "iso_country_codes = [\"GB\", \"FR\", \"JP\"]\n",
    "total_population = \"SP.POP.TOTL\"\n",
    "\n",
    "dataset = api.get_dataset(total_population, iso_country_codes, date=\"2010:2012\")\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``IndicatorDataset`` instance contains the direct API response and various metadata. Use ``dataset.as_dict()`` to return a tidy dictionary of the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'FR': {'2012': 65659809.0, '2011': 65342780.0, '2010': 65027507.0},\n",
       " 'GB': {'2012': 63700215.0, '2011': 63258810.0, '2010': 62766365.0},\n",
       " 'JP': {'2012': 127629000.0, '2011': 127833000.0, '2010': 128070000.0}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.as_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some examples of the metadata available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://api.worldbank.org/v2/countries/GBR;FRA;JPN/indicators/SP.POP.TOTL?date=2010%3A2012&format=json&per_page=10000'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.api_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Population, total'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.indicator_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/indicator/SP.POP.TOTL?format=json&per_page=10000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'id': '19', 'value': 'Climate Change'}, {'id': '8', 'value': 'Health '}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.indicator_topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'FR': 'France', 'GB': 'United Kingdom', 'JP': 'Japan'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.countries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to create your own data structures, you can process the raw API response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'page': 1,\n",
       "  'pages': 1,\n",
       "  'per_page': 10000,\n",
       "  'total': 9,\n",
       "  'sourceid': '2',\n",
       "  'lastupdated': '2020-07-01'},\n",
       " [{'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'FR', 'value': 'France'},\n",
       "   'countryiso3code': 'FRA',\n",
       "   'date': '2012',\n",
       "   'value': 65659809,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'FR', 'value': 'France'},\n",
       "   'countryiso3code': 'FRA',\n",
       "   'date': '2011',\n",
       "   'value': 65342780,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'FR', 'value': 'France'},\n",
       "   'countryiso3code': 'FRA',\n",
       "   'date': '2010',\n",
       "   'value': 65027507,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'GB', 'value': 'United Kingdom'},\n",
       "   'countryiso3code': 'GBR',\n",
       "   'date': '2012',\n",
       "   'value': 63700215,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'GB', 'value': 'United Kingdom'},\n",
       "   'countryiso3code': 'GBR',\n",
       "   'date': '2011',\n",
       "   'value': 63258810,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'GB', 'value': 'United Kingdom'},\n",
       "   'countryiso3code': 'GBR',\n",
       "   'date': '2010',\n",
       "   'value': 62766365,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'JP', 'value': 'Japan'},\n",
       "   'countryiso3code': 'JPN',\n",
       "   'date': '2012',\n",
       "   'value': 127629000,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'JP', 'value': 'Japan'},\n",
       "   'countryiso3code': 'JPN',\n",
       "   'date': '2011',\n",
       "   'value': 127833000,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0},\n",
       "  {'indicator': {'id': 'SP.POP.TOTL', 'value': 'Population, total'},\n",
       "   'country': {'id': 'JP', 'value': 'Japan'},\n",
       "   'countryiso3code': 'JPN',\n",
       "   'date': '2010',\n",
       "   'value': 128070000,\n",
       "   'unit': '',\n",
       "   'obs_status': '',\n",
       "   'decimal': 0}]]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.api_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for indicators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't always know what indicators we want to use, so we can search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/indicator?format=json&per_page=10000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1591"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "population_indicators = api.get_indicators(search=\"population\")\n",
    "len(population_indicators)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ah. That's not a very manageable number. The API returns over 8000 indicator codes, and lots of them have \"population\" in the title. Luckily, most of those indicators don't really have much data, so we can forget about them. You can browse the indicators with the best data coverage at http://data.worldbank.org/indicator, and you can pass ``common_only=True`` to throw away all indicators that aren't included on that page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/indicator?format=json&per_page=10000\n",
      "There are now only 246 indicators to browse!\n"
     ]
    }
   ],
   "source": [
    "population_indicators = api.get_indicators(search=\"population\", common_only=True)\n",
    "print(\"There are now only %d indicators to browse.\" % len(population_indicators))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't want to print that many results in the documentation, so let's filter some more. The API query string parameters are directly mapped to kwargs for each method. For the ``get_indicators`` method, this means we can filter by topic or source:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/topic/8/indicator?format=json&per_page=10000\n",
      "We've narrowed it down to 109 indicators.\n"
     ]
    }
   ],
   "source": [
    "health_topic_id = 8\n",
    "health_indicators = api.get_indicators(search=\"population\", common_only=True, topic=health_topic_id)\n",
    "print(\"We've narrowed it down to %d indicators.\" % len(health_indicators))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each indicator has a variety of metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('SH.DYN.AIDS.FE.ZS',\n",
      " {'name': \"Women's share of population ages 15+ living with HIV (%)\",\n",
      "  'source': {'id': '2', 'value': 'World Development Indicators'},\n",
      "  'sourceNote': 'Prevalence of HIV is the percentage of people who are '\n",
      "                'infected with HIV. Female rate is as a percentage of the '\n",
      "                'total population ages 15+ who are living with HIV.',\n",
      "  'sourceOrganization': 'UNAIDS estimates.',\n",
      "  'topics': [{'id': '8', 'value': 'Health '}, {'id': '17', 'value': 'Gender'}],\n",
      "  'unit': ''})\n"
     ]
    }
   ],
   "source": [
    "pprint(list(health_indicators.items())[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That data might be useful, but it's not very friendly if you just want to grab some API codes. If that's what you want, you can pass the results to the ``print_codes`` method:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/indicator?format=json&per_page=10000\n",
      "SH.TBS.CURE.ZS                 Tuberculosis treatment success rate (% of new cases)\n",
      "SH.TBS.DOTS                    Tuberculosis cases detected under DOTS (%)\n",
      "SH.TBS.DTEC.ZS                 Tuberculosis case detection rate (%, all forms)\n",
      "SH.TBS.INCD                    Incidence of tuberculosis (per 100,000 people)\n",
      "SH.TBS.INCD.HG                 Incidence of tuberculosis, high uncertainty bound (per 100,000 people)\n",
      "SH.TBS.INCD.LW                 Incidence of tuberculosis, low uncertainty bound (per 100,000 people)\n",
      "SH.TBS.MORT                    Tuberculosis death rate (per 100,000 people)\n",
      "SH.TBS.MORT.HG                 Deaths due to tuberculosis among HIV-negative people, high uncertainty bound (per 100,000 population)\n",
      "SH.TBS.MORT.LW                 Deaths due to tuberculosis among HIV-negative people, low uncertainty bound (per 100,000 population)\n",
      "SH.TBS.PREV                    Tuberculosis prevalence rate (per 1000,000 population, WHO)\n",
      "SH.TBS.PREV.HG                 Tuberculosis prevalence rate, high uncertainty bound (per 1000,000 population, WHO)\n",
      "SH.TBS.PREV.LW                 Tuberculosis prevalence rate, low uncertainty bound (per 1000,000 population, WHO)\n"
     ]
    }
   ],
   "source": [
    "api.print_codes(api.get_indicators(search=\"tuberculosis\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are ``get_`` functions matching all API endpoints (countries, regions, sources, etc.), and the ``search`` parameter and ``print_codes`` method can be used on any of them. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/country?format=json&per_page=10000\n",
      "AE                             United Arab Emirates\n",
      "GB                             United Kingdom\n",
      "US                             United States\n"
     ]
    }
   ],
   "source": [
    "countries = api.get_countries(search=\"united\")\n",
    "api.print_codes(countries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More searching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're not sure what to search for, just leave out the ``search`` parameter. By default, the ``get_`` methods return all API results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/region?format=json&per_page=10000\n",
      "http://api.worldbank.org/v2/source?format=json&per_page=10000\n",
      "There are 48 regions and 61 sources.\n"
     ]
    }
   ],
   "source": [
    "all_regions = api.get_regions()\n",
    "all_sources = api.get_sources()\n",
    "\n",
    "print(\"There are %d regions and %d sources.\" % (len(all_regions), len(all_sources)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``search`` parameter actually just calls a ``search_results`` method, which you can use directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'20': {'code': 'PSD',\n",
      "        'concepts': '3',\n",
      "        'dataavailability': 'Y',\n",
      "        'description': '',\n",
      "        'lastupdated': '2020-07-07',\n",
      "        'metadataavailability': 'Y',\n",
      "        'name': 'Quarterly Public Sector Debt',\n",
      "        'url': ''},\n",
      " '22': {'code': 'QDS',\n",
      "        'concepts': '3',\n",
      "        'dataavailability': 'Y',\n",
      "        'description': '',\n",
      "        'lastupdated': '2020-04-30',\n",
      "        'metadataavailability': 'Y',\n",
      "        'name': 'Quarterly External Debt Statistics SDDS',\n",
      "        'url': ''},\n",
      " '23': {'code': 'QDG',\n",
      "        'concepts': '3',\n",
      "        'dataavailability': 'Y',\n",
      "        'description': '',\n",
      "        'lastupdated': '2020-04-30',\n",
      "        'metadataavailability': 'Y',\n",
      "        'name': 'Quarterly External Debt Statistics GDDS',\n",
      "        'url': ''},\n",
      " '54': {'code': 'JED',\n",
      "        'concepts': '3',\n",
      "        'dataavailability': 'Y',\n",
      "        'description': '',\n",
      "        'lastupdated': '2020-06-04',\n",
      "        'metadataavailability': '',\n",
      "        'name': 'Joint External Debt Hub',\n",
      "        'url': ''},\n",
      " '6': {'code': 'IDS',\n",
      "       'concepts': '3',\n",
      "       'dataavailability': 'Y',\n",
      "       'description': '',\n",
      "       'lastupdated': '2019-12-02',\n",
      "       'metadataavailability': 'Y',\n",
      "       'name': 'International Debt Statistics',\n",
      "       'url': ''}}\n"
     ]
    }
   ],
   "source": [
    "pprint(api.search_results(\"debt\", all_sources))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the ``search`` parameter only searches the title of an entity (eg. a country name, or source title). If you want to search all fields, set the ``search_full`` flag to ``True``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://api.worldbank.org/v2/topic?format=json&per_page=10000\n",
      "http://api.worldbank.org/v2/topic?format=json&per_page=10000\n",
      "1 topic(s) match(es) 'poverty' in the title field, and 8 topics match 'poverty' in all fields.\n"
     ]
    }
   ],
   "source": [
    "narrow_matches = api.get_topics(search=\"poverty\")\n",
    "wide_matches = api.get_topics(search=\"poverty\", search_full=True)\n",
    "\n",
    "print(\"%d topic(s) match(es) 'poverty' in the title field, and %d topics match 'poverty' in all fields.\" % (len(narrow_matches), len(wide_matches)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API options "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All endpoint query string parameters are directly mapped to method kwargs. Different \n",
    "kwargs are available for each ``get_`` method (documented in the method's docstring).\n",
    "\n",
    "\n",
    "* **language:**     ``EN``, ``ES``, ``FR``, ``AR`` or ``ZH``. Non-English languages seem to have less info in the responses.\n",
    "\n",
    "* **date:**          String formats - ``2001``, ``2001:2006``, ``2003M01:2004M06``, ``2005Q2:2005Q4``. Replace the years with your own. Not all indicators have monthly or quarterly data.\n",
    "\n",
    "* **mrv:**           Most recent value, ie. ``mrv=3`` returns the three most recent values for an indicator.\n",
    "\n",
    "* **gapfill:**       ``Y`` or ``N``. If using an MRV value, fills missing values with the next available value (I think tracking back as far as the MRV value allows). Defaults to ``N``.\n",
    "\n",
    "* **frequency:**     Works with MRV, can specify quarterly (``Q``), monthly (``M``) or yearly (``Y``). Not all indicators have monthly and quarterly data. \n",
    "\n",
    "* **source:**        ID number to filter indicators by data source.\n",
    "\n",
    "* **topic:**         ID number to filter indicators by their assigned category. Cannot give both source and topic in the same request.\n",
    "\n",
    "* **incomelevel:**   List of 3-letter IDs to filter results by income level category.\n",
    "\n",
    "* **lendingtype:**   List of 3-letter IDs to filter results by lending type. \n",
    "\n",
    "* **region:**        List of 3-letter IDs to filter results by region.\n",
    "\n",
    "If no date or MRV value is given, **MRV defaults to 1**, returning the most recent\n",
    "value.\n",
    "\n",
    "Any given kwarg that is not in the above list will be directly added to the query\n",
    "string, eg. ``foo=\"bar\"`` will add ``&foo=bar`` to the URL. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Country codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``wbpy`` supports ISO 1366 alpha-2 and alpha-3 country codes. The World Bank uses some non-ISO 2-letter and 3-letter codes for regions, which are also supported. You can access them via the ``NON_STANDARD_REGIONS`` attribute, which returns a dictionary of codes and region info. Again, to see the codes, pass the dictionary to the ``print_codes`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1A                             Arab World\n",
      "1W                             World\n",
      "4E                             East Asia & Pacific (developing only)\n",
      "7E                             Europe & Central Asia (developing only)\n",
      "8S                             South Asia\n",
      "A4                             Sub-Saharan Africa excluding South Africa\n",
      "A5                             Sub-Saharan Africa excluding South Africa and Nigeria\n",
      "A9                             Africa\n",
      "C4                             East Asia and the Pacific (IFC classification)\n",
      "C5                             Europe and Central Asia (IFC classification)\n",
      "C6                             Latin America and the Caribbean (IFC classification)\n",
      "C7                             Middle East and North Africa (IFC classification)\n",
      "C8                             South Asia (IFC classification)\n",
      "C9                             Sub-Saharan Africa (IFC classification)\n",
      "EU                             European Union\n",
      "JG                             Channel Islands\n",
      "KV                             Kosovo\n",
      "M2                             North Africa\n",
      "OE                             OECD members\n",
      "S1                             Small states\n",
      "S2                             Pacific island small states\n",
      "S3                             Caribbean small states\n",
      "S4                             Other small states\n",
      "XC                             Euro area\n",
      "XD                             High income\n",
      "XE                             Heavily indebted poor countries (HIPC)\n",
      "XJ                             Latin America & Caribbean (developing only)\n",
      "XL                             Least developed countries: UN classification\n",
      "XM                             Low income\n",
      "XN                             Lower middle income\n",
      "XO                             Low & middle income\n",
      "XP                             Middle income\n",
      "XQ                             Middle East & North Africa (developing only)\n",
      "XR                             High income: nonOECD\n",
      "XS                             High income: OECD\n",
      "XT                             Upper middle income\n",
      "XU                             North America\n",
      "XY                             Not classified\n",
      "Z4                             East Asia & Pacific (all income levels)\n",
      "Z7                             Europe & Central Asia (all income levels)\n",
      "ZF                             Sub-Saharan Africa (developing only)\n",
      "ZG                             Sub-Saharan Africa (all income levels)\n",
      "ZJ                             Latin America & Caribbean (all income levels)\n",
      "ZQ                             Middle East & North Africa (all income levels)\n"
     ]
    }
   ],
   "source": [
    "api.print_codes(api.NON_STANDARD_REGIONS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Climate API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two methods to the climate API - ``get_modelled``, which returns a ``ModelledDataset`` instance, and ``get_instrumental``, which returns an ``InstrumentalDataset`` instance. The World Bank API has multiple date pairs associated with each dataset, but a single ``wbpy`` call will make multiple API calls and return all the dates associated with the requested data type.\n",
    "\n",
    "For full explanation of the data and associated models, see the [Climate API documentation](http://data.worldbank.org/developers/climate-data-api). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like the Indicators API, locations can be ISO-1366 alpha-2 or alpha-3 country codes. They can also be IDs corresponding to regional river basins. A basin map can be found in the official Climate API documentation. The API includes a KML interface that returns basin definitions, but this is currently not supported by ``wbpy``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instrumental data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The available arguments and their definitions are accessible via the ``ARG_DEFINITIONS`` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pr': 'Precipitation (rainfall and assumed water equivalent), in millimeters',\n",
       " 'tas': 'Temperature, in degrees Celsius'}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_api = wbpy.ClimateAPI()\n",
    "\n",
    "c_api.ARG_DEFINITIONS[\"instrumental_types\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['year', 'month', 'decade']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_api.ARG_DEFINITIONS[\"instrumental_intervals\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<wbpy.climate.InstrumentalDataset({'tas': 'Temperature, in degrees Celsius'}, 'decade') with id: 140420664386392>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iso_and_basin_codes = [\"AU\", 1, 302]\n",
    "\n",
    "dataset = c_api.get_instrumental(data_type=\"tas\", interval=\"decade\", locations=iso_and_basin_codes)\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``InstrumentalDataset`` instance stores the API responses, various metadata and methods for accessing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'1': {'1960': 5.975941,\n",
      "       '1970': 6.1606956,\n",
      "       '1980': 6.3607564,\n",
      "       '1990': 6.600332,\n",
      "       '2000': 7.3054743},\n",
      " '302': {'1960': -12.850627,\n",
      "         '1970': -12.679074,\n",
      "         '1980': -12.295782,\n",
      "         '1990': -11.440549,\n",
      "         '2000': -11.460049},\n",
      " 'AU': {'1900': 21.078014,\n",
      "        '1910': 21.296726,\n",
      "        '1920': 21.158426,\n",
      "        '1930': 21.245909,\n",
      "        '1940': 21.04456,\n",
      "        '1950': 21.136906,\n",
      "        '1960': 21.263151,\n",
      "        '1970': 21.306032,\n",
      "        '1980': 21.633171,\n",
      "        '1990': 21.727072,\n",
      "        '2000': 21.741446,\n",
      "        '2010': 21.351604}}\n"
     ]
    }
   ],
   "source": [
    "pprint(dataset.as_dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tas': 'Temperature, in degrees Celsius'}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.data_type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modelled data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``get_modelled`` returns data derived from Global Glimate Models. There are various possible data types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tmin_means': 'Average daily minimum temperature, Celsius',\n",
       " 'tmax_means': 'Average daily maximum temperature, Celsius',\n",
       " 'tmax_days90th': \"Number of days with max temperature above the control period's 90th percentile (hot days)\",\n",
       " 'tmin_days90th': \"Number of days with min temperature above the control period's 90th percentile (warm nights)\",\n",
       " 'tmax_days10th': \"Number of days with max temperature below the control period's 10th percentile (cool days)\",\n",
       " 'tmin_days10th': \"Number of days with min temperature below the control period's 10th percentile (cold nights)\",\n",
       " 'tmin_days0': 'Number of days with min temperature below 0 degrees Celsius',\n",
       " 'ppt_days': 'Number of days with precipitation > 0.2mm',\n",
       " 'ppt_days2': 'Number of days with precipitation > 2mm',\n",
       " 'ppt_days10': 'Number of days with precipitation > 10mm',\n",
       " 'ppt_days90th': \"Number of days with precipitation > the control period's 90th percentile\",\n",
       " 'ppt_dryspell': 'Average number of days between precipitation events',\n",
       " 'ppt_means': 'Average daily precipitation',\n",
       " 'pr': 'Precipitation (rainfall and assumed water equivalent), in millimeters',\n",
       " 'tas': 'Temperature, in degrees Celsius'}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_api.ARG_DEFINITIONS[\"modelled_types\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mavg': 'Monthly average',\n",
       " 'annualavg': 'Annual average',\n",
       " 'manom': 'Average monthly change (anomaly).',\n",
       " 'annualanom': 'Average annual change (anomaly).',\n",
       " 'aanom': 'Average annual change (anomaly).',\n",
       " 'aavg': 'Annual average'}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_api.ARG_DEFINITIONS[\"modelled_intervals\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<wbpy.climate.ModelledDataset({'pr': 'Precipitation (rainfall and assumed water equivalent), in millimeters'}, {'annualavg': 'Annual average'}) with id: 140420644546936>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "locations = [\"US\"]\n",
    "modelled_dataset = c_api.get_modelled(\"pr\", \"aavg\", locations)\n",
    "modelled_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``as_dict()`` method for ``ModelledDataset`` takes a kwarg to specify the SRES used for future values. The API uses the A2 and B1 scenarios:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'bccr_bcm2_0': {'US': {'1939': 790.6361028238144,\n",
      "                        '1959': 780.0266445283039,\n",
      "                        '1979': 782.7526463724754,\n",
      "                        '1999': 785.2701232986692,\n",
      "                        '2039': 783.1710625360416,\n",
      "                        '2059': 804.3092939039038,\n",
      "                        '2079': 804.6334514665734,\n",
      "                        '2099': 859.8239942059615}},\n",
      " 'cccma_cgcm3_1': {'US': {'1939': 739.3362184367556,\n",
      "                          '1959': 746.2975320411192,\n",
      "                          '1979': 739.4449188917432,\n",
      "                          '1999': 777.7889471267924,\n",
      "                          '2039': 808.1474524518724,\n",
      "                          '2059': 817.1428223416907,\n",
      "                          '2079': 841.7569757399672,\n",
      "                          '2099': 871.6962130920673}},\n",
      " 'cnrm_cm3': {'US': {'1939': 939.7243516499025,\n",
      "                     '1959': 925.6653938577782,\n",
      "                     '1979': 940.2236730711822,\n",
      "                     '1999': 947.5967851291585,\n",
      "                     '2039': 962.6036875622598,\n",
      "                     '2059': 964.4556538112397,\n",
      "                     '2079': 970.7166949721155,\n",
      "                     '2099': 987.7517843651068}},\n",
      " 'csiro_mk3_5': {'US': {'1939': 779.0404023054358,\n",
      "                        '1959': 799.5361627973773,\n",
      "                        '1979': 796.607564873811,\n",
      "                        '1999': 798.381580457504,\n",
      "                        '2039': 843.0498166357976,\n",
      "                        '2059': 867.6557574566958,\n",
      "                        '2079': 884.6635096827529,\n",
      "                        '2099': 914.4892749739001}},\n",
      " 'ensemble_10': {'US': {'1939': 666.6475434339079,\n",
      "                        '1959': 665.7610790034265,\n",
      "                        '1979': 667.1738791525539,\n",
      "                        '1999': 670.415327533486,\n",
      "                        '2039': 686.4924376146926,\n",
      "                        '2059': 690.3005736391768,\n",
      "                        '2079': 693.0003564697117,\n",
      "                        '2099': 709.0425715268083}},\n",
      " 'ensemble_50': {'US': {'1939': 850.8566502216561,\n",
      "                        '1959': 851.1821259381916,\n",
      "                        '1979': 852.9435213996902,\n",
      "                        '1999': 855.0129391106861,\n",
      "                        '2039': 873.0523341457085,\n",
      "                        '2059': 880.9922361302446,\n",
      "                        '2079': 892.9013887250998,\n",
      "                        '2099': 916.5180306375303}},\n",
      " 'ensemble_90': {'US': {'1939': 1020.5076048129349,\n",
      "                        '1959': 1018.0491512612145,\n",
      "                        '1979': 1020.2880850240846,\n",
      "                        '1999': 1029.4064082957505,\n",
      "                        '2039': 1048.7391596386938,\n",
      "                        '2059': 1056.5504828474266,\n",
      "                        '2079': 1067.6845781511777,\n",
      "                        '2099': 1106.7227445303276}},\n",
      " 'gfdl_cm2_0': {'US': {'1939': 898.1444407247458,\n",
      "                       '1959': 890.578762482606,\n",
      "                       '1979': 873.31199204601,\n",
      "                       '1999': 890.4286021472773,\n",
      "                       '2039': 884.667792836329,\n",
      "                       '2059': 891.2301658572712,\n",
      "                       '2079': 858.2037683045394,\n",
      "                       '2099': 862.2664763719782}},\n",
      " 'gfdl_cm2_1': {'US': {'1939': 847.0485774775588,\n",
      "                       '1959': 832.6677468315708,\n",
      "                       '1979': 840.3616008806812,\n",
      "                       '1999': 827.3124179982142,\n",
      "                       '2039': 854.7964182636986,\n",
      "                       '2059': 870.5118615966802,\n",
      "                       '2079': 868.5767216101426,\n",
      "                       '2099': 878.4820392256858}},\n",
      " 'ingv_echam4': {'US': {'1939': 845.4780955327558,\n",
      "                        '1959': 845.2359494710544,\n",
      "                        '1979': 852.7707911085288,\n",
      "                        '1999': 851.9327652092476,\n",
      "                        '2039': 866.0409073675132,\n",
      "                        '2059': 872.7481665480419,\n",
      "                        '2079': 900.9028488881945,\n",
      "                        '2099': 919.2062848249728}},\n",
      " 'inmcm3_0': {'US': {'1939': 825.6505057699028,\n",
      "                     '1959': 844.9800055068362,\n",
      "                     '1979': 860.5045147370352,\n",
      "                     '1999': 843.0909232427455,\n",
      "                     '2039': 877.4836079129254,\n",
      "                     '2059': 885.5902710722888,\n",
      "                     '2079': 878.6926405756873,\n",
      "                     '2099': 895.3363280260298}},\n",
      " 'ipsl_cm4': {'US': {'1939': 897.1020362453344,\n",
      "                     '1959': 881.2890852171191,\n",
      "                     '1979': 888.57049309408,\n",
      "                     '1999': 900.6203651333254,\n",
      "                     '2039': 911.0684866203087,\n",
      "                     '2059': 908.9880107774133,\n",
      "                     '2079': 901.9352518210636,\n",
      "                     '2099': 924.6232749957305}},\n",
      " 'miroc3_2_medres': {'US': {'1939': 815.9899280956733,\n",
      "                            '1959': 820.924517871823,\n",
      "                            '1979': 820.561522790526,\n",
      "                            '1999': 819.1997264378206,\n",
      "                            '2039': 815.5123964532938,\n",
      "                            '2059': 812.3150259004544,\n",
      "                            '2079': 810.515112232343,\n",
      "                            '2099': 817.447065795786}},\n",
      " 'miub_echo_g': {'US': {'1939': 815.7217424350092,\n",
      "                        '1959': 819.1216945126766,\n",
      "                        '1979': 816.4814506968534,\n",
      "                        '1999': 836.9998036334464,\n",
      "                        '2039': 841.4617194083404,\n",
      "                        '2059': 847.7322521257802,\n",
      "                        '2079': 880.5316551949228,\n",
      "                        '2099': 920.7048218268357}},\n",
      " 'mpi_echam5': {'US': {'1939': 932.4105818597735,\n",
      "                       '1959': 930.0013750415483,\n",
      "                       '1979': 921.4702739003415,\n",
      "                       '1999': 941.6353488835641,\n",
      "                       '2039': 969.6867904854836,\n",
      "                       '2059': 990.3857663124111,\n",
      "                       '2079': 1000.6110341746332,\n",
      "                       '2099': 1080.5289311209049}},\n",
      " 'mri_cgcm2_3_2a': {'US': {'1939': 728.5749928767182,\n",
      "                           '1959': 720.3172590678807,\n",
      "                           '1979': 732.943309679262,\n",
      "                           '1999': 727.9981579483319,\n",
      "                           '2039': 735.1725461582992,\n",
      "                           '2059': 751.6773914898702,\n",
      "                           '2079': 776.7754868580876,\n",
      "                           '2099': 798.3133892715804}},\n",
      " 'ukmo_hadcm3': {'US': {'1939': 839.9996105395489,\n",
      "                        '1959': 849.9134671410114,\n",
      "                        '1979': 851.505705112856,\n",
      "                        '1999': 848.5821514937204,\n",
      "                        '2039': 874.371671909573,\n",
      "                        '2059': 877.512058895459,\n",
      "                        '2079': 881.875457040721,\n",
      "                        '2099': 927.3730832143624}},\n",
      " 'ukmo_hadgem1': {'US': {'1939': 841.7922922262945,\n",
      "                         '1959': 845.698748695459,\n",
      "                         '1979': 834.3090961483945,\n",
      "                         '1999': 831.8516144217097,\n",
      "                         '2039': 866.4876927782285,\n",
      "                         '2059': 864.5861500956854,\n",
      "                         '2079': 882.1356350906877,\n",
      "                         '2099': 907.0139017841842}}}\n"
     ]
    }
   ],
   "source": [
    "pprint(modelled_dataset.as_dict(sres=\"a2\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, various metadata is available, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bccr_bcm2_0': 'BCM 2.0',\n",
       " 'cccma_cgcm3_1': 'CGCM 3.1 (T47)',\n",
       " 'cnrm_cm3': 'CNRM CM3',\n",
       " 'csiro_mk3_5': 'CSIRO Mark 3.5',\n",
       " 'gfdl_cm2_0': 'GFDL CM2.0',\n",
       " 'gfdl_cm2_1': 'GFDL CM2.1',\n",
       " 'ingv_echam4': 'ECHAM 4.6',\n",
       " 'inmcm3_0': 'INMCM3.0',\n",
       " 'ipsl_cm4': 'IPSL-CM4',\n",
       " 'miub_echo_g': 'ECHO-G',\n",
       " 'mpi_echam5': 'ECHAM5/MPI-OM',\n",
       " 'mri_cgcm2_3_2a': 'MRI-CGCM2.3.2',\n",
       " 'ukmo_hadcm3': 'UKMO HadCM3',\n",
       " 'ukmo_hadgem1': 'UKMO HadGEM1',\n",
       " 'ensemble_90': '90th percentile values of all models together',\n",
       " 'ensemble_10': '10th percentile values of all models together',\n",
       " 'ensemble_50': '50th percentile values of all models together'}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "modelled_dataset.gcms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('1920', '1939'),\n",
       " ('1940', '1959'),\n",
       " ('1960', '1979'),\n",
       " ('1980', '1999'),\n",
       " ('2020', '2039'),\n",
       " ('2040', '2059'),\n",
       " ('2060', '2079'),\n",
       " ('2080', '2099')]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "modelled_dataset.dates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default cache function uses system temporary files. You can specify your own. The function has to take a url, and return the corresponding web page as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def func(url):\n",
    "    # Basic function that doesn't do any caching\n",
    "    from six.moves.urllib import request\n",
    "    return request.urlopen(url).read()\n",
    "\n",
    "# Either pass it in on instantiation...\n",
    "ind_api = wbpy.IndicatorAPI(fetch=func)\n",
    "\n",
    "# ...or point api.fetch to it. \n",
    "climate_api = wbpy.ClimateAPI()\n",
    "climate_api.fetch = func"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}