incubator/mixture-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gaussian Mixtures\n",
    "\n",
    "Sometimes, our data look like they are generated by a \"mixture\" model. What do we mean by that? In statistics land, it means we believe that there are \"mixtures\" of subpopulations generating the data that we observe. A common activity, then, is to estimate the subpopulation parameters.\n",
    "\n",
    "Let's take a look at it by generating some simulated data to illustrate the point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.executable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pymc3 as pm\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by first generating a mixture distribution that is composed of unit width Gaussians (i.e. $ N(\\mu, 1) $) that are slightly overlapping.\n",
    "\n",
    "$$ pop \\sim GaussianMixture(\\mu=[0, 3], \\sigma=[1, 1]) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_mixture_data(mus, sizes):\n",
    "    \"\"\"\n",
    "    Generates mixture data\n",
    "    \"\"\"\n",
    "    subpop1 = np.random.normal(loc=mus[0], scale=1, size=sizes[0])\n",
    "    subpop2 = np.random.normal(loc=mus[1], scale=1, size=sizes[1])\n",
    "    mixture = np.concatenate([subpop1, subpop2])\n",
    "    return mixture\n",
    "\n",
    "\n",
    "mixture = generate_mixture_data(mus=[0, 3], sizes=[20000, 20000])\n",
    "plt.hist(mixture, bins=100)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to reiterate the point, one of the Gaussian distributions has a mean at 0, and the other has a mean at 3. Both subpopulations are present in equal proportions in the larger population, i.e. they have equal weighting.\n",
    "\n",
    "Let's see if we can use PyMC3 to recover those parameters. Since we know that there are two mixture components, we can encode this in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pm.Model() as model:\n",
    "    mu = pm.Cauchy(\"mu\", alpha=1, beta=1, shape=(2,))\n",
    "    sd = pm.HalfCauchy(\"sd\", beta=1, shape=(2,))\n",
    "\n",
    "    w = pm.Dirichlet(\"w\", a=np.array([1, 1]))  # mixture component weights. See below!\n",
    "\n",
    "    like = pm.NormalMixture(\"like\", w=w, mu=mu, sd=sd, observed=mixture)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with model:\n",
    "    trace = pm.sample(2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm.traceplot(trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, sometimes, in our final population, one sub-population is present at a lower frequency than the other sub-population. Let's try to simulate that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mixture = generate_mixture_data(\n",
    "    mus=[0, 3], sizes=[20000, 2000]\n",
    ")  # One is at 1/10 the size of the other.\n",
    "plt.hist(mixture, bins=100)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pm.Model() as model:\n",
    "    mu = pm.Cauchy(\"mu\", alpha=1, beta=1, shape=(2,))\n",
    "    sd = pm.HalfCauchy(\"sd\", beta=1, shape=(2,))\n",
    "\n",
    "    w = pm.Dirichlet(\"w\", a=np.array([1, 1]))  # mixture component weights. See below!\n",
    "\n",
    "    like = pm.NormalMixture(\"like\", w=w, mu=mu, sd=sd, observed=mixture)\n",
    "\n",
    "with model:\n",
    "    trace = pm.sample(2000)\n",
    "    pm.traceplot(trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is really good. We have fewer samples for the group with $ \\mu = 3 $, which thus means that we are much less confident about the value of $ \\mu $ and $ \\sigma $. What's neat is that we are nonetheless equally confident of the relative weighting of the two groups: one is much smaller in proportion than the other!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generalized Mixtures\n",
    "\n",
    "We used Gaussian (a.k.a. Normal) distributions for generating the data. However, what if the data didn't come from a Gaussian distribution, but instead came from two Poissons?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_poisson_mixtures(lams, sizes):\n",
    "    grp1 = np.random.poisson(lam=lams[0], size=sizes[0])\n",
    "    grp2 = np.random.poisson(lam=lams[1], size=sizes[1])\n",
    "\n",
    "    mixture = np.concatenate([grp1, grp2])\n",
    "    return mixture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mixture = generate_poisson_mixtures(lams=[14, 30], sizes=[1000, 250])\n",
    "plt.hist(mixture)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pm.Model() as model:\n",
    "    lam = pm.Exponential(\"lam\", lam=1, shape=(2,))\n",
    "    components = pm.Poisson.dist(\n",
    "        mu=lam, shape=(2,)\n",
    "    )  # must use dist, not plain Poisson object!\n",
    "\n",
    "    w = pm.Dirichlet(\"w\", a=np.array([1, 1]))  # mixture component weights. See below!\n",
    "\n",
    "    like = pm.Mixture(\"like\", w=w, comp_dists=components, observed=mixture)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with model:\n",
    "    trace = pm.sample(2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm.traceplot(trace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm.energyplot(trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It worked! There was one minor detail that I had to learn from Junpeng Lao, who answered [my question](https://discourse.pymc.io/t/is-there-an-example-on-how-to-work-with-generalized-mixture-models/726) on the PyMC3 discourse site. That detail is this - that we have to use the `pm.Poisson.dist(...)` syntax, rather than `pm.Poisson(...)` syntax.\n",
    "\n",
    "Now, what if we had much fewer data points? How would our confidence levels change?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mixture = generate_poisson_mixtures(lams=[14, 30], sizes=[100, 25])\n",
    "plt.hist(mixture)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pm.Model() as model:\n",
    "    lam = pm.Exponential(\"lam\", lam=1, shape=(2,))\n",
    "    components = pm.Poisson.dist(\n",
    "        mu=lam, shape=(2,)\n",
    "    )  # must use dist, not plain Poisson object!\n",
    "\n",
    "    w = pm.Dirichlet(\"w\", a=np.array([1, 1]))  # mixture component weights. See below!\n",
    "\n",
    "    like = pm.Mixture(\"like\", w=w, comp_dists=components, observed=mixture)\n",
    "\n",
    "    trace = pm.sample(2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm.traceplot(trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At ~100-ish data points, it's still not too hard to tell. What if we had fewer data points?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mixture = generate_poisson_mixtures(lams=[14, 30], sizes=[10, 2])\n",
    "plt.hist(mixture)\n",
    "plt.show()\n",
    "\n",
    "with pm.Model() as model:\n",
    "    lam = pm.Exponential(\"lam\", lam=1, shape=(2,))\n",
    "    components = pm.Poisson.dist(\n",
    "        mu=lam, shape=(2,)\n",
    "    )  # must use dist, not plain Poisson object!\n",
    "\n",
    "    w = pm.Dirichlet(\"w\", a=np.array([1, 1]))  # mixture component weights. See below!\n",
    "\n",
    "    like = pm.Mixture(\"like\", w=w, comp_dists=components, observed=mixture)\n",
    "\n",
    "    trace = pm.sample(2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm.traceplot(trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model identifiability problems come in to play. `lam` parameters are very hard to estimate with little data. Moral of the story - get more data :)\n",
    "\n",
    "Let's try **Weibull distribution mixtures**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_weibull_mixture(alphas, betas, size):\n",
    "\n",
    "    wb1 = betas[0] * np.random.weibull(a=alphas[0], size=size)\n",
    "    wb2 = betas[1] * np.random.weibull(a=alphas[1], size=size)\n",
    "\n",
    "    mixture = np.concatenate([wb1, wb2])\n",
    "\n",
    "    return mixture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mixture = generate_weibull_mixture(alphas=[10, 10], betas=[20, 10], size=20000)\n",
    "plt.hist(mixture)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pm.Model():\n",
    "    x = pm.Weibull(\"wb\", alpha=1, beta=100).random()\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pm.Model() as model:\n",
    "    # alpha1 = pm.HalfNormal('alpha1', sd=10)\n",
    "    # beta1 = pm.HalfNormal('beta1', sd=10)\n",
    "    # comp1 = pm.Weibull.dist(alpha=alpha1, beta=beta1)\n",
    "\n",
    "    # alpha2 = pm.HalfNormal('alpha2', sd=10)\n",
    "    # beta2 = pm.HalfNormal('beta2', sd=10)\n",
    "    # comp2 = pm.Weibull.dist(alpha=alpha2, beta=beta2)\n",
    "\n",
    "    alpha = pm.HalfNormal(\"alpha\", sd=10, shape=(2,))\n",
    "    beta = pm.HalfNormal(\"beta\", sd=10, shape=(2,))\n",
    "\n",
    "    components = pm.Weibull.dist(alpha=alpha, beta=beta, shape=(2,))\n",
    "\n",
    "    w = pm.Dirichlet(\"w\", a=np.array([1, 1]))\n",
    "\n",
    "    likelihood = pm.Mixture(\"likelihood\", comp_dists=components, w=w, observed=mixture)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with model:\n",
    "    trace = pm.sample(2000, init=\"advi\", njobs=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm.traceplot(trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parameters are recovered! It was hard-won, though; sampling takes a long time with NUTS (but we get very good samples), and I had to experiment with ADVI init vs. auto init before empirically finding out that ADVI init works better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bayesian",
   "language": "python",
   "name": "bayesian"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}