strings-regular-expressions-and-text-data-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Strings, Regular Expressions, and Text Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Suppose you would like to construct a data set of short definitions and facts drawn from a large text corpus such as the set of all article abstracts found in the English-language edition of Wikipedia. You would also like the workflow you assemble to be reusable and flexible, allowing you to update the data set and/or enhance its capabilities quickly. What Python features and libraries, both built-in and from the broader community, can you use to accomplish this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "While built-in string methods have limited flexibility and regular expressions have limited expressive power, both can still be leveraged in creative ways to implement scalable workflows that process and analyze text data. This article explores these tools and introduces a few useful peripheral packages and techniques within the context of the use case described above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Retrieving and Reading the Raw Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Archives containing all abstracts found in the English-language edition of Wikipedia are regularly updated and available at [https://dumps.wikimedia.org/enwiki/latest/](https://dumps.wikimedia.org/enwiki/latest/) (a link you can find via [this landing page](https://en.wikipedia.org/wiki/Wikipedia:Database_download)). You decide to use the consolidated abstracts archive file containing all of the abstracts. How can you retrieve it in an automated manner? One way to retrieve a large file that is available online while also getting user-friendly feedback about progress is to use the [requests](https://requests.readthedocs.io/en/master/) library together with the [tqdm](https://pypi.org/project/tqdm/) library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The tqdm library makes it possible to display an animated progress bar while an instance of an [iterable](https://docs.python.org/3/glossary.html#term-iterable) data structure emits each of its elements. The example below illustrates the output. The `position` and `leave` parameters ensure that the progress bar is not interrupted by newline characters if the output is being displayed within Jupyter Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████| 3/3 [00:03<00:00,  1.01s/it]\n"
     ]
    }
   ],
   "source": [
    "import tqdm\n",
    "import time\n",
    "for i in tqdm.tqdm(range(3), position=0, leave=True):\n",
    "    time.sleep(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The [requests](https://requests.readthedocs.io/en/master/) library makes it possible to create an iterable stream of the data from a file that is located at a specified URL by setting the [`stream`](https://requests.readthedocs.io/en/master/user/advanced/#body-content-workflow) parameter to `True`. You can then retrieve the data in chunks and [write them to a file](https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files) while displaying progress to the user. In the example below, the `total` parameter supplied to `tqdm.tqdm` is the number of 1 MB (*i.e.*, $1024^2$ byte) chunks of data in the file; it is determined by dividing file size indicated in the HTTP [response header](https://requests.readthedocs.io/en/master/user/quickstart/#response-headers) by the chosen `chunk_size` of 1 MB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████| 10/10 [00:06<00:00,  1.66it/s]\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from math import ceil\n",
    "\n",
    "url = \"https://python.supply/\"\n",
    "url += \"strings-regular-expressions-and-text-data-analysis/\"\n",
    "url += \"enwiki-20191120-abstract-sample.xml.gz\"\n",
    "\n",
    "with requests.get(url, stream=True) as response:\n",
    "    file_size = int(response.headers['Content-length'])\n",
    "    chunk_size = 1024*1024\n",
    "    with open(\"abstracts.xml.gz\", \"wb\") as file:\n",
    "        chunks = response.iter_content(chunk_size=chunk_size)\n",
    "        for chunk in tqdm.tqdm(\n",
    "                chunks, total=ceil(file_size/chunk_size),\n",
    "                position=0, leave=True\n",
    "            ):\n",
    "            file.write(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Once you have the compressed file `abstracts.xml.gz` on your disk, you can open it and iterate over the data by uncompressing it one line at a time using the built-in [gzip](https://docs.python.org/3/library/gzip.html) library. Below, the total number of lines in the file is counted using a comprehension and the built-in [`sum`](https://docs.python.org/3/library/functions.html#sum) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "238564\n"
     ]
    }
   ],
   "source": [
    "import gzip\n",
    "with gzip.open(\"abstracts.xml.gz\") as file:\n",
    "    print(sum(1 for line in file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Processing Raw String Data with String Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Given that the data set is in XML format, the most general approach to parsing it is to use an XML parser such as [lxml](https://pypi.org/project/lxml/). However, in this case the text content in the file follows a consistent format: the text for each abstract appears on its own line in its entirety, surrounded by the two element delimiters `<abstract>` and `</abstract>`. Thus, you can iterate over every line in the file and determine whether or not you want to extract it. You divide this workflow into two steps: extracting the lines, and then extracting the abstracts themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First, you decide to implement a [generator](https://docs.python.org/3/glossary.html#term-generator) that iterates over all the lines in the file and yields them one at a time. In case the text data on a particular line has an unknown encoding, the [chardet](https://pypi.org/project/chardet/) library is used to check each line. This is not likely to be an issue in this particular data set, but allowing only those lines that are valid [ASCII](https://en.wikipedia.org/wiki/ASCII) or [UTF-8](https://en.wikipedia.org/wiki/UTF-8) can help you avoid any unexpected issues later in the process. Before yielding it, you convert each line from binary data into a string using the [`decode`](https://docs.python.org/3/library/stdtypes.html#bytes.decode) method of the binary data type [`bytes`](https://docs.python.org/3/library/stdtypes.html#bytes) and then remove any whitespace at the beginning or end of the string using [`strip`](https://docs.python.org/3/library/stdtypes.html#str.strip)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from chardet import detect\n",
    "def lines():\n",
    "    with gzip.open(\"abstracts.xml.gz\") as file:\n",
    "        for line in file:\n",
    "            if detect(line)[\"encoding\"] in ['ascii', 'utf-8']:\n",
    "                yield line.decode().strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How do you extract the actual text of an abstract from each line of the form `<abstract>...</abstract>`? You can take advantage of the [`startswith`](https://docs.python.org/3/library/stdtypes.html#str.startswith) and [`endswith`](https://docs.python.org/3/library/stdtypes.html#str.endswith) built-in string methods to identify these lines, and then you can use string [slice indices](https://docs.python.org/3/tutorial/introduction.html?highlight=string%20slice#strings) to extract the raw text of each abstract. The function below defines a [generator](https://docs.python.org/3/glossary.html#term-generator) that yields every abstract that appears on any line in the compressed file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def abstracts():\n",
    "    for line in lines():\n",
    "        if line.startswith(\"<abstract>\") and\\\n",
    "           line.endswith(\"</abstract>\"):\n",
    "            line = line[len(\"<abstract>\"):-len(\"</abstract>\")]\n",
    "            yield line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can use the [`islice`](https://docs.python.org/3/library/itertools.html#itertools.islice) function for iterable data structure instances to iterate over only a portion of the abstracts. The line below calculates the maximum length of the first 100 abstracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "553"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from itertools import islice\n",
    "max(len(a) for a in islice(abstracts(), 0, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Extracting Strings with a Specific Structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "[Regular expressions](https://en.wikipedia.org/wiki/Regular_expression) are mathematical constructs that can be used to describe *sets of strings* that follow a particular pattern or format. The history and conceptual details of regular expressions are far beyond the scope of this article, but you only need to understand a few basic building blocks to begin using the built-in [re](https://docs.python.org/3/library/re.html) library to identify when a string adheres to a specific pattern."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In Python, regular expressions are represented as strings. In the example below, the regular expression `\"abc\"` is satisfied only by those strings that begin with the exact sequence of characters `abc`. To check whether a string satisfies a regular expression, it is sufficient to compile the regular expression string using [`compile`](https://docs.python.org/3/library/re.html#re.compile) and then to use the [`match`](https://docs.python.org/3/library/re.html#re.match) method to perform the check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "re.compile(\"abc\").match(\"abcd\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To ensure that only an entire string (as opposed to only a portion of the string) can satisfy a regular expression, it is possible to add `^` to the beginning of the regular expression and `$` to the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 3), match='abc'>\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(re.compile(\"^abc$\").match(\"abc\"))\n",
    "print(re.compile(\"^abc$\").match(\"abcd\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The [syntax for regular expression strings](https://docs.python.org/3/library/re.html#regular-expression-syntax) consists of a large collection of special characters and character sequences. For the purposes of the motivating example, the below syntactic constructs are sufficient:\n",
    "* `[a-z]` is satisfied by any lowercase single letter;\n",
    "* `\\s` is satisfied by any single whitespace character and `\\.` is satisfied by a period;\n",
    "* given any regular expression `r`, the set of strings that satisfy `r` also satisfy `(r)` and vice versa;\n",
    "* given any regular expression `r`, the set of strings that satisfy `r*` consists of all strings that are concatenations of zero or more individual strings that each independently satisfy `r` (*e.g.*, `\"abc\"`, `\"aaaa\"`, and `\"\"` all satisfy `[a-z]*`);\n",
    "* given any two regular expressions `r` and `s`, the regular expression `r|s` is satisfied by any string that satisfies either `r`, `s`, or both;\n",
    "* given any two regular expressions `r` and `s`, the regular expression `rs` is satisfied by any string consisting of a portion that satisfies `r` followed immediately by a portion that satisfies `s` (*e.g.*, `\"xy\"` satisfies `[a-z][a-z]`, while `\"x\"` and `\"xyz\"` do not). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To make it easier to experiment quickly with larger and more complex regular expressions while reusing common building blocks, you can define functions that take smaller expressions and build larger ones. Because regular expressions are represented using Python strings, you can use string methods such as [`join`](https://docs.python.org/3/library/stdtypes.html#str.join) to accomplish this. The functions below also use [argument unpacking syntax](https://docs.python.org/3/tutorial/controlflow.html#arbitrary-argument-lists) to make code that invokes them more concise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def atom(s):\n",
    "    # Wrap an expression in parenthesis.\n",
    "    return \"(\" + s + \")\"\n",
    "\n",
    "def options(*rs):\n",
    "    # Given expressions r1, r2, ..., rN, build\n",
    "    # the expression (r1)|(r2)|...|(rN).\n",
    "    return atom(\"|\".join([atom(r) for r in rs]))\n",
    "\n",
    "def adjacent(*rs):\n",
    "    # Given expressions r1, r2, ..., rN, build\n",
    "    # the expression (r1)(r2)...(rN).\n",
    "    return atom(\"\".join([atom(r) for r in rs]))\n",
    "\n",
    "def spaced(*rs):\n",
    "    # Given expressions r1, r2, ..., rN, build\n",
    "    # the expression (r1)\\s(r2)\\s...\\s(rN).\n",
    "    return atom(\"\\s\".join([atom(r) for r in rs]))\n",
    "\n",
    "def repeated(r):\n",
    "    # Given the expression r0, build ((r0)*).\n",
    "    return atom(atom(r) + \"*\")\n",
    "\n",
    "def repeated_spaced(r):\n",
    "    # Given the expression r0, build (((r0)(\\s))*(r0)).\n",
    "    return adjacent(repeated(adjacent(r, \"\\s\")), r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The definitions below are fairly self-explanatory and use the above functions to build more complex regular expressions. These are useful as building blocks for building a regular expression that matches a sentence of the form you are trying to identify in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "article = options(\"A\", \"a\", \"An\", \"an\", \"The\", \"the\")\n",
    "word = atom(\"[a-z]+\")\n",
    "verb = options(\"are\", \"is\")\n",
    "period = atom(\"\\.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The definition below is for a regular expression that is satisfied by any string that (1) begins with a definite or indefinite article, (2) has any non-zero number of words following the article, (3) has a linking verb followed by one or more additional words, and (4) ends with a period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = adjacent(spaced(article, word, repeated_spaced(word), verb, repeated_spaced(word)), period)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can see the actual string that the function invocations above construct. It should be apparent that assembling (or modifying) a string such as the one below manually in its raw form is likely to be an error-prone process. Abstractions such as those above are an effective way to manage complexity in regular expressions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'((((((A)|(a)|(An)|(an)|(The)|(the)))\\\\s(([a-z]+))\\\\s(((((((([a-z]+))(\\\\s)))*))(([a-z]+))))\\\\s(((are)|(is)))\\\\s(((((((([a-z]+))(\\\\s)))*))(([a-z]+))))))((\\\\.)))'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You decide to use this regular expression to identify facts and definitions in the abstracts, storing a reusable compiled version of it in as `sentence_re` for better performance. Before you check whether an abstract matches the regular expression `sentence`, you use the string method [`split`](https://docs.python.org/3/library/stdtypes.html#str.split) to separate the string into a list of substrings (*i.e.*, those substrings that are found between the instances of the `.` character in the string). You then use list index notation to keep only the first entry. Since the `split` method does not include the character used to split the string, you need to concatenate it back onto the first substring. The example below applies this workflow to a portion of the abstracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An audio file format is a file format for storing digital audio data on a computer system.\n",
      "The bell curve is typical of the normal distribution.\n",
      "The immune system is a host defense system comprising many biological structures and processes within an organism that protects against disease.\n",
      "A monolithic kernel is an operating system architecture where the entire operating system is working in kernel space.\n"
     ]
    }
   ],
   "source": [
    "sentence_re = re.compile(\"^\" + sentence + \"$\")\n",
    "for a in islice(abstracts(), 0, 400):\n",
    "    a = a.split(\".\")[0] + '.' # Keep only first sentence.\n",
    "    if sentence_re.match(a) is not None:\n",
    "        print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic Analysis Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You are interested in learning more about the particular terms or phrases being described by the first sentence of each abstract. In particular, you would like to know the distribution of the number of words before the verb. One approach is to use the [`search`](https://docs.python.org/3/library/re.html#re.search) method to find the first instance of a verb and then to inspect the contents of the [Match object](https://docs.python.org/3/library/re.html#match-objects) to determine the location of the match. Then it is possible to use the string [`count`](https://docs.python.org/3/library/stdtypes.html#str.count) method on the substring (up to and including that matched verb) to determine the number of spaces (and thus, words) before the first verb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 210777/210777 [06:05<00:00, 576.85it/s]\n"
     ]
    }
   ],
   "source": [
    "counts = []\n",
    "verb_re = re.compile(verb)\n",
    "for abstract in tqdm.tqdm(\n",
    "        abstracts(), position=0, leave=True, total=210777\n",
    "    ):\n",
    "    abstract = abstract.split(\".\")[0] + '.'\n",
    "    if sentence_re.match(abstract) is not None:\n",
    "        if (m := verb_re.search(abstract)) is not None:\n",
    "            counts.append(abstract[:m.end()].count(\" \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You may notice in the above example the use of an [assignment expression](https://www.python.org/dev/peps/pep-0572/) involving the `:=` operator. Scenarios such as this one play a significant role in the rationale for this new feature (introduced in Python 3.8). The alternative syntax would have required a separate line for the statement `m = verb_re.search(abstract)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now that the number of words before the verb in each abstract is in the list of integers `counts`, it is straightforward to generate a histogram using [Matplotlib](https://matplotlib.org/). The results suggest that the most common case is one involving three words before the verb (which, given that the initial [definite or indefinite article](https://en.wikipedia.org/wiki/Article_(grammar)) is also being counted, suggests that most abstracts refer to a concept described by a two-word phrase)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAdKUlEQVR4nO3de7gcVZnv8e+PcA8XiQQMJBLQoIijASNwBkQQRbyBHLzAEYUBxcEg4HVARsXx4YwX1PHoAUUBM8hFZgQNXokMCI5CSELIhcCYgQQ2CUkUhQjHaJL3/LHWLpqd3ntXV+/q3tn793mefrqqutaqt3v37rdqVdVaigjMzMwAtuh2AGZmNnw4KZiZWcFJwczMCk4KZmZWcFIwM7PClt0OoB277rprTJ48udthmJltVubOnfu7iBjf7LXNOilMnjyZOXPmdDsMM7PNiqTl/b1WW/ORpEmSbpW0RNJiSefk5RdKelTS/Px4Y0OZ8yUtlfSApNfXFZuZmTVX55HCeuAjETFP0o7AXEmz8mtfiYiLG1eW9BLgRGB/YA/gF5L2jYgNNcZoZmYNajtSiIiVETEvT68FlgB7DlDkOOC6iFgXEQ8BS4GD6orPzMw21ZGrjyRNBg4A7sqLzpK0QNIVknbJy/YEHmko1sPAScTMzIZY7UlB0g7A94FzI+JJ4FLgBcBUYCXwpd5VmxTfpGMmSWdImiNpzpo1a2qK2sxsdKo1KUjaipQQro6IGwAiYlVEbIiIjcC3eKaJqAeY1FB8IrCib50RcVlETIuIaePHN72iyszMKqrz6iMBlwNLIuLLDcsnNKx2PLAoT88ETpS0jaS9gSnA7LriMzOzTdV59dGhwLuBhZLm52WfAE6SNJXUNLQMeD9ARCyWdD1wH+nKpem+8sjMrLNqSwoR8Suanyf4yQBlLgIuqismMzMb2GZ9R7Mlb3rd4azo6fcGxVL2mLgXP551+xBFZGabKyeFEWBFz3JuOrO9k+5vubS9pGJmI4N7STUzs4KTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7OCk4KZmRWcFMzMrOCkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs4KTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWqC0pSJok6VZJSyQtlnROXj5O0ixJv83PuzSUOV/SUkkPSHp9XbGZmVlzdR4prAc+EhH7AYcA0yW9BDgPuCUipgC35HnyaycC+wPHAJdIGlNjfGZm1kdtSSEiVkbEvDy9FlgC7AkcB8zIq80A3pqnjwOui4h1EfEQsBQ4qK74zMxsUx05pyBpMnAAcBewe0SshJQ4gN3yansCjzQU68nL+tZ1hqQ5kuasWbOmzrDNzEad2pOCpB2A7wPnRsSTA63aZFlssiDisoiYFhHTxo8fP1RhmpkZNScFSVuREsLVEXFDXrxK0oT8+gRgdV7eA0xqKD4RWFFnfGZm9mx1Xn0k4HJgSUR8ueGlmcApefoU4IcNy0+UtI2kvYEpwOy64jMzs01tWWPdhwLvBhZKmp+XfQL4HHC9pNOBh4G3A0TEYknXA/eRrlyaHhEbaozPzMz6qC0pRMSvaH6eAOCofspcBFxUV0xmZjYw39FsZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7PCoElB0qGSxubpkyV9WdJe9YdmZmadVuZI4VLgaUkvBz4OLAf+tdaozMysK8okhfUREaSurb8aEV8Fdqw3LDMz64YydzSvlXQ+cDJweB74Zqt6wzIzs24oc6TwTmAdcHpEPEYa4+CLtUZlZmZdUeZI4UMR8Q+9MxHxsKT9a4zJzMy6pMyRwuuaLHvDUAdiZmbd1++RgqQzgQ8AL5C0oOGlHYFf1x2YmZl13kDNR9cAPwX+GTivYfnaiHi81qjMzKwr+m0+iognImIZ8FXg8YhYHhHLgb9KOrhTAZqZWeeUvXntTw3zT+VlZmY2wpRJCso3rwEQERupdxhPMzPrkjJJ4UFJZ0vaKj/OAR6sOzAzM+u8Mknh74G/BR4FeoCDgTPqDMrMzLpj0GagiFgNnNiBWMzMrMsGTQqStgVOB/YHtu1dHhGn1RiXmZl1QZnmo6uA5wGvB34JTATW1hmUmZl1R5mk8MKI+CTwVETMAN4E/E29YZmZWTeUSQp/zc9/lPRSYGdgcm0RmZlZ15S53+AySbsA/wjMBHYAPllrVGZm1hUDJgVJWwBPRsQfgNuBfToSlZmZdcWAzUf57uWzOhSLmZl1WZlzCrMkfVTSJEnjeh+1R2ZmZh1X5pxC7/0I0xuWBW5KMjMbccokhf0i4s+NC/INbWZmNsKUaT5qNsqaR14zMxuBBhqO83nAnsB2kg4AlF/aCdi+A7GZmVmHDdR89HrgVFK3Fl/imaSwFvjEYBVLugJ4M7A6Il6al10IvA9Yk1f7RET8JL92PqmPpQ3A2RHx8xbfi5mZtanfpJC7tJgh6YSI+H6Fur8DfB341z7LvxIRFzcukPQSUk+s+wN7AL+QtG9EbKiwXTMzq6jMOYWJknZS8m1J8yQdPVihiLgdeLxkHMcB10XEuoh4CFgKHFSyrJmZDZEySeG0iHgSOBrYDfg74HNtbPMsSQskXZG7z4B07uKRhnV68rJNSDpD0hxJc9asWdNsFTMzq6jUGM35+Y3AlRFxb8OyVl0KvACYCqwknato3EajaLKMiLgsIqZFxLTx48dXDMPMzJopkxTmSrqZlBR+LmlHYGOVjUXEqojYkLvP+BbPNBH1AJMaVp0IrKiyDTMzq65MUjgdOA94ZUQ8DWxNakJqmaQJDbPHA4vy9EzgREnbSNobmALMrrINMzOrrswYzRslTQZOlhTAryLixsHKSboWOALYVVIP8GngCElTSU1Dy4D3520slnQ9cB+wHpjuK4/MzDqvzBjNlwAvBK7Ni94v6bURMX2AYkTESU0WXz7A+hcBFw0Wj5mZ1adM30evBl4aEQEgaQawsNaozMysK8qcU3gAeH7D/CRgQT3hmJlZNw3U99FNpLb/nYElkmbn+YNxh3hmZiPSQM1HFw/wmpmZjUAD9X30y04GYmZm3TfoOQVJh0i6W9KfJP1F0gZJT3YiODMz66wyJ5q/DpwE/BbYDnhvXmZmZiNMmUtSiYilksbkG8qulOQTzWZmI1CZpPC0pK2B+ZK+QOrIbmy9YZmZWTeUaT56d17vLOAp0n0KJ9QZlJmZdUeZvo+W58k/A5+pNxwzM+umMkcKZmY2SjgpmJlZod+kIOmq/HxO58IxM7NuGuhI4RWS9gJOk7SLpHGNj04FaGZmnTPQieZvAD8D9gHm8uxxlCMvNzOzEaTfI4WI+D8RsR9wRUTsExF7NzycEMzMRqAyl6SeKenlwKvyotsjwuMpmJmNQGU6xDsbuBrYLT+ulvTBugMzM7POK9PNxXuBgyPiKQBJnwd+A3ytzsDMzKzzytynIGBDw/wGnn3S2czMRogyRwpXAndJujHPvxW4vL6QzMysW8qcaP6ypNuAw0hHCH8XEffUHZiZmXVe2fEU5gHzao7FzMy6zH0fmZlZwUnBzMwKAyYFSWMk/aJTwZiZWXcNmBTymMxPS9q5Q/GYmVkXlTnR/GdgoaRZpOE4AYiIs2uLyszMuqJMUvhxfpiZ2QhX5j6FGZK2A54fEQ90ICYzM+uSMh3ivQWYTxpbAUlTJc2sOzAzM+u8MpekXggcBPwRICLmA3vXGJOZmXVJmaSwPiKe6LMs6gjGzMy6q0xSWCTpfwFjJE2R9DXg14MVknSFpNWSFjUsGydplqTf5uddGl47X9JSSQ9Ien2ld2NmZm0pkxQ+COwPrAOuBZ4Ezi1R7jvAMX2WnQfcEhFTgFvyPJJeApyYt3MMcImkMSW2YWZmQ2jQpBART0fEBcBRwJERcUFE/LlEuduBx/ssPg6YkadnkLrh7l1+XUSsi4iHgKWk8xhmZtZBZa4+eqWkhcAC0k1s90p6RcXt7R4RKwHy8255+Z7AIw3r9eRlzeI5Q9IcSXPWrFlTMQwzM2umTPPR5cAHImJyREwGppMG3hlKzUZya3oyOyIui4hpETFt/PjxQxyGmdnoViYprI2IO3pnIuJXwNqK21slaQJAfl6dl/cAkxrWmwisqLgNMzOrqN+kIOlASQcCsyV9U9IRkl4t6RLgtorbmwmckqdPAX7YsPxESdtI2huYAsyuuA0zM6tooG4uvtRn/tMN04PepyDpWuAIYFdJPbn854DrJZ0OPAy8HSAiFku6HrgPWA9Mzz20mplZB/WbFCLiyHYqjoiT+nnpqH7Wvwi4qJ1tmplZewbtEE/Sc4D3AJMb13fX2WZmI0+ZrrN/AtwJLAQ21huOmZl1U5mksG1EfLj2SMzMrOvKXJJ6laT3SZqQ+y4aJ2lc7ZGZmVnHlTlS+AvwReACnrnqKIB96grKzMy6o0xS+DDwwoj4Xd3BmJlZd5VpPloMPF13IGZm1n1ljhQ2APMl3UrqPhvwJalmZiNRmaTwg/wwM7MRbtCkEBEzBlvHzMxGhjJ3ND9Ek76OIsJXH5mZjTBlmo+mNUxvS+rEzvcpmJmNQGWG4/x9w+PRiPgX4DUdiM3MzDqsTPPRgQ2zW5COHHasLSIzM+uaMs1HjeMqrAeWAe+oJRozM+uqMlcftTWugpmZbT7KNB9tA5zApuMp/FN9YZmZWTeUaT76IfAEMJeGO5rNzGzkKZMUJkbEMbVHYmZmXVemQ7xfS/qb2iMxM7OuK3OkcBhwar6zeR0gICLiZbVGZmZmHVcmKbyh9iis6x57bBUH7LdXW3XsMXEvfjzr9iGKyMy6ocwlqcs7EYh1V2xcz01njm+rjrdc6q+K2eauzDkFMzMbJZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7OCk4KZmRWcFMzMrFCm76MhJ2kZsBbYAKyPiGmSxgHfIw3mswx4R0T8oRvxmZmNVt08UjgyIqZGxLQ8fx5wS0RMAW7J82Zm1kHDqfnoOGBGnp4BvLWLsZiZjUrdSgoB3CxprqQz8rLdI2IlQH7erVlBSWdImiNpzpo1azoUrpnZ6NCVcwrAoRGxQtJuwCxJ95ctGBGXAZcBTJs2LeoK0MxsNOrKkUJErMjPq4EbgYOAVZImAOTn1d2IzcxsNOt4UpA0VtKOvdPA0cAiYCZwSl7tFOCHnY7NzGy060bz0e7AjZJ6t39NRPxM0t3A9ZJOBx4G3t6F2MzMRrWOJ4WIeBB4eZPlvweO6nQ8Zmb2jOF0SaqZmXWZk4KZmRWcFMzMrOCkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZoVvDcVr2ptcdzoqe5W3VsXrVKmD80ARkZqOak0KXrehZzk1ntveDPu2Tjw5RNGY22rn5yMzMCk4KZmZWcPORDZnHHlvFAfvt1VYde0zcix/Pun2IIjKzVjkp2JCJjevbPj/ylkvbO+luZu1x85GZmRWcFMzMrOCkYGZmBScFMzMr+ERzG3w38tDzFUxm3eWk0AbfjTz0fAWTWXe5+cjMzAo+UrARx01QZtU5KdiI4yYos+rcfGRmZgUnBTMzKzgpmJlZwecUzJoYipPVj//hj4zb5Tlt1eET3tZpTgpmTQzFyeppn3yUmz4xpa06fMLbOm3YJQVJxwBfBcYA346Iz3U5JLOu8eW11mnDKilIGgP8X+B1QA9wt6SZEXFfdyMz6w5fXjv8DEX3NsM5UQ+rpAAcBCyNiAcBJF0HHAc4KZhV1O7RxnD+AWvVUPVXdteFL22rjuGcqBUR3Y6hIOltwDER8d48/27g4Ig4q2GdM4Az8uyLgAfa2OSuwO+6WN51uI7NoY7hEIPrGNo69oqIpoegw+1IQU2WPStrRcRlwGVDsjFpTkRM61Z51+E6Noc6hkMMrqOeOpoZbvcp9ACTGuYnAiu6FIuZ2agz3JLC3cAUSXtL2ho4EZjZ5ZjMzEaNYdV8FBHrJZ0F/Jx0SeoVEbG4xk222ww1FM1YrsN1DPc6hkMMrqOeOjYxrE40m5lZdw235iMzM+siJwUzMyuMyqQg6RhJD0haKum8CuWvkLRa0qI2Ypgk6VZJSyQtlnROhTq2lTRb0r25js9UjGWMpHsk/ahK+VzHMkkLJc2XNKdiHc+R9O+S7s+fy/9osfyL8vZ7H09KOrfFOj6UP8tFkq6VtG1r7wIknZPLLy67/WbfKUnjJM2S9Nv8vEuFOt6e49goadDLF/up44v5b7JA0o2SBuzlr586PpvLz5d0s6Q9Wq2j4bWPSgpJu1aI40JJjzZ8R95YJQ5JH8y/IYslfaFCHN9riGGZpPkV6pgq6c7e/zlJBw1UR2kRMaoepBPY/w3sA2wN3Au8pMU6DgcOBBa1EccE4MA8vSPwXxXiELBDnt4KuAs4pEIsHwauAX7UxvtZBuza5t9mBvDePL018Jw2/86PkW7SKVtmT+AhYLs8fz1waovbfSmwCNiedCHHL4ApVb5TwBeA8/L0ecDnK9SxH+kmz9uAaRXjOBrYMk9/vmIcOzVMnw18o9U68vJJpAtRlg/2fesnjguBj7bw92xWx5H577pNnt+tyntpeP1LwKcqxHEz8IY8/Ubgtla+q/09RuORQtGVRkT8BejtSqO0iLgdeLydICJiZUTMy9NrgSWkH6VW6oiI+FOe3So/WrpyQNJE4E3At1spN9Qk7UT64l8OEBF/iYg/tlHlUcB/R0Sr/QlsCWwnaUvSD3ur98nsB9wZEU9HxHrgl8DxgxXq5zt1HClRkp/f2modEbEkIkrf9d9PHTfn9wJwJ+n+oVbreLJhdiyDfE8H+B/7CvDxwcoPUkdp/dRxJvC5iFiX11ldNQ5JAt4BXFuhjgB2ytM7M0T3dI3GpLAn8EjDfA8t/hgPNUmTgQNIe/qtlh2TDz1XA7MiotU6/oX0T7ax1W33EcDNkuYqdUXSqn2ANcCVuSnr25LGthHPiQzyj9ZXRDwKXAw8DKwEnoiIm1vc7iLgcEnPlbQ9aQ9u0iBl+rN7RKzMsa0EdqtYz1A6DfhplYKSLpL0CPAu4FMVyh8LPBoR91bZfoOzclPWFYM1yfVjX+BVku6S9EtJr2wjllcBqyLitxXKngt8MX+mFwPntxFHYTQmhUG70ugkSTsA3wfO7bM3VUpEbIiIqaS9t4Mkle6pS9KbgdURMbfV7TZxaEQcCLwBmC7p8BbLb0k6PL40Ig4AniI1mbRM6cbHY4F/a7HcLqS9872BPYCxkk5upY6IWEJqYpkF/IzUPLl+wEKbCUkXkN7L1VXKR8QFETEplz9rsPX7bHt74AIqJJM+LgVeAEwlJf4vVahjS2AX4BDgY8D1eY+/ipNoceelwZnAh/Jn+iHyUXa7RmNSGDZdaUjaipQQro6IG9qpKze13AYc00KxQ4FjJS0jNaO9RtJ3K25/RX5eDdxIaqZrRQ/Q03Ck8++kJFHFG4B5EbGqxXKvBR6KiDUR8VfgBuBvW914RFweEQdGxOGkQ/4qe4EAqyRNAMjPAzZT1EnSKcCbgXdFbsRuwzXACS2WeQEpWd+bv68TgXmSntdKJRGxKu9IbQS+RevfU0jf1Rty8+1s0lH2gCe9m8lNlP8T+F6FGABOIX1HIe0ADcmJ5tGYFIZFVxp5z+JyYElEfLliHeN7rwSRtB3pR+3+suUj4vyImBgRk0mfw39EREt7xnnbYyXt2DtNOjHZ0pVZEfEY8IikF+VFR1G9y/Sqe18PA4dI2j7/fY4inetpiaTd8vPzSf/0VfcEZ5L+8cnPP6xYT1uUBr76B+DYiHi6Yh2NQ9AdSwvfU4CIWBgRu0XE5Px97SFdqPFYi3FMaJg9nha/p9kPgNfk+vYlXRRRpbfS1wL3R0RPhbKQdmZfnadfQ/Wdj2cbirPVm9uD1M77X6SrkC6oUP5a0qHnX0lfztMr1HEYqdlqATA/P97YYh0vA+7JdSxikCsYBqnrCCpefUQ6H3Bvfiyu8pnmeqYCc/L7+QGwS4U6tgd+D+xcMYbPkH6wFgFXka8wabGOO0gJ7V7gqKrfKeC5wC2kf/ZbgHEV6jg+T68DVgE/r1DHUtJ5uN7v6WBXDjWr4/v5M10A3ATs2WodfV5fxuBXHzWL4ypgYY5jJjChQh1bA9/N72ce8Joq7wX4DvD3bXw/DgPm5u/ZXcArqnzn+z7czYWZmRVGY/ORmZn1w0nBzMwKTgpmZlZwUjAzs4KTgpmZFZwUbNiRdFuZHj2HYDtnK/XGWukO3Qrbu1DSR0uue6qkr7dY/9vz+7m1WoT1yT2BtnyDl3XesBqO06xdkraMZzpvG8wHSL1MPlRDHGMiYsNQ1zuI04EPRESppNDiZ1WZpDF1b8OGjo8UrBJJk/Ne6bdyn/I357uqn7WnL2nX3C1B797vDyTdJOkhSWdJ+nDuAO9OSeMaNnGypF8rjUtwUC4/Nndidncuc1xDvf8m6SZSd8J9Y/1wrmeR8vgGkr5BuulupqQP9Vn/J5JelqfvkfSpPP1ZSe9V8sVc30JJ78yvH6E0RsY1pBukkHSBUr/7vyB1Yd27jbMl3Zc7Zruun495kqSf5fKfbih7stI4GvMlfVOpU8RPkW5m+kaObVtJV+b47pF0ZH+flaSP5c90gZqMySHpTDWMGZDr+Fp/seTlf5L0T5LuAnrHxfhYXne2pBf2856t24biDjg/Rt8DmEzqHG1qnr8eODlP30buu5/UJ8yyPH0q6e7YHYHxwBPkOzpJXSKf21D+W3n6cHIf8sD/btjGc0h3pY/N9fbQ5I5f4BWkH+ixwA6kO64PyK8to8ldsaSO+KaTuiW+m3wXMHAr6Yf9BFKHd2OA3UndY0wg3RX+FLB3n21vn+taSu7Ln9RFQW9//JuMG5Hf00rSXc3bke6enUbqmvsmYKu83iXAe5p87h8BrszTL84xbtv3syJ1SXIZqaPILYAfAYf3iWU8qbv53vmfkhLQQLEE8I6GMsvId7oD76GNsTv8qPfhIwVrx0MR0Tti1FxSohjMrRGxNiLWkJLCTXn5wj7lr4WiH/mdlPp4Oho4T6mr8NtIP3LPz+vPiohmfdYfBtwYEU9FGnviBlJ3xQO5g5SMDgN+DOyg1Evn5EhjExwGXBupY7VVpDETertPnh3PNEe9Km/76Ug94Db2sbUAuFqpF9b+mnBmRcTvI+L/5bgPI/XH9Arg7vw5HEU64mn2vq8CiIj7SYPS7NtQb+9ndXR+3EPqsuHFQGM/ReS/1YOSDpH0XFJi/M9BYtlA6tqi0bUNzy2Nqmed43MK1o51DdMbSHu0kH7kenc4+g5n2VhmY8P8Rp79fezb/0qQ9mZPiD6Dxkg6mLSH3kyVLo3vJu2VP0g6ItgVeB8p8Q1WZ984+utH5k2kxHMs8ElJ+8em7fv9fQYzImKwvvPLxijgnyPim4PU9z3SYDD3kxJdSBoolj/HpudUop9pG0Z8pGB1WEbagwR4W8U6etvpDyMNdvMEaRjGD+YfIyQdUKKe24G3KvV8OpbUQdwdAxWINCLfI6QfwTvz+h9tKHc78M7clj+e9OM+u59tHy9pO6VeZN+S494CmBTphPDHSU1hOzQp/zqlcZq3I4269p+kjvHepmd6Yh0naa9+tv2uvM6+pCOqZiOw/Rw4TWlcDyTt2Vt3HzfkGE7ima6ey8bS650Nz78ZYD3rIh8pWB0uJg088m7gPyrW8QdJvya1xZ+Wl32WNFLcgpwYlpH6+O9XRMyT9B2e+dH+dkTcU2L7d5B6OH1a0h2k/vt7k8KNpOaPe0l7vB+PiMckvbjJtr9H6ll0eUP5McB3Je1M2lP/SjQfevRXpCagFwLXRMQcAEn/SBrlbgtSr5nTc/2NLiGddF5IOnI7NSLWqc9YMBFxs6T9gN/k1/4EnEyfsRsi4g+S7iONIz47L7uvZCy9tsknnrcgJRcbhtxLqpmZFdx8ZGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMDOzgpOCmZkV/j8jLedBkQwuOwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.hist(\n",
    "    counts,\n",
    "    bins=range(max(counts)), density=False, histtype=\"bar\",\n",
    "    color=\"#DD7600\", edgecolor=\"k\", alpha=0.8\n",
    ")\n",
    "plt.ylabel(\"number of abstracts\")\n",
    "plt.xlabel(\"number of words before verb\")\n",
    "plt.xticks(range(0, max(counts)))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Further Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This article reviews a use case while referencing or employing techniques that are drawn from a broad array of disciplines and topic areas: data science (specificially, [data cleaning](https://en.wikipedia.org/wiki/Data_cleansing)), [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load), parsing and data structure search and traversal, text data analysis and natural language processing, regular expressions and regular language grammars, web scraping, and a few others. Providing more detailed directions for exploring each of these further is beyond the scope of this article, but a few points that relate specifically to implementing such workflows in Python are worth noting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The [regex](https://pypi.org/project/regex/) package is backwards-compatible with the built-in [re](https://docs.python.org/3/library/re.html) library but offers additional features such as fuzzy matching. The long-lived [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) library is described as a tool for parsing HTML and XML documents (typically in scenarios involving web scraping), but is actually a powerful text data processing tool in its own right. An assortment of sophisticated natural language processing tools and techniques are available in the [Natural Language Toolkit](https://www.nltk.org/), including the ability to tokenize, tag, and parse text. Finally, towards the machine learning end of the spectrum are libraries like [Gensim](https://radimrehurek.com/gensim/), which provide capabilities such as [topic modeling](https://en.wikipedia.org/wiki/Topic_model)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}