examples/Examples-SGF2020-Paper-Simple_and_Efficient_Bootstrap_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple and Efficient Bootstrap Validation of Predictive Models Using SAS/STAT® Software\n",
    "\n",
    "## Example File for Invited Paper &nbsp;&bullet;&nbsp;  SAS Global Forum 2020\n",
    "\n",
    "This notebook contains an executable version of the running example in the paper at https://github.com/saspy-bffs/sgf-2020-bootstrap-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 1\n",
    "\n",
    "The code below creates macro variables encapsulating model parameters. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n",
    "\n",
    "**Note**: You may also wish to change the values of the macro variables in order to explore bootstrap validation for different models. The running example is adapted from Module 10 at https://wwwn.cdc.gov/nchs/nhanes/tutorials/samplecode.aspx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%let response_variable_condition = bpxsar >= 140 OR bpxdar >= 90 OR bpq050a = 1;\n",
    "%let response_variable = hyper;\n",
    "%let outcome = &response_variable.(EVENT='1');\n",
    "%let class_variables = bpq100d dmq051 dmd110;\n",
    "%let predictor_variables = lbxtc bpq100d bmxbmi ridageyr lbxtr dmq051 dmd110 indhhinc indfmpir;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 2\n",
    "\n",
    "The code below downloads the example dataset for Module 10 at https://wwwn.cdc.gov/nchs/nhanes/tutorials/samplecode.aspx. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename tempfile \"%sysfunc(pathname(work))/analysis_data.sas7bdat\";\n",
    "proc http\n",
    "        url='https://wwwn.cdc.gov/nchs/data/tutorials/analysis_data.sas7bdat'\n",
    "        method='get'\n",
    "        out=tempfile\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 3\n",
    "\n",
    "The code below subsets the dataset downloaded above to rows with no missing values for the predicator variables, as well as creating a response variable. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data example_dataset;\n",
    "    set analysis_data;\n",
    "    \n",
    "    * Create outcome variable;\n",
    "    if (&response_variable_condition.) then &response_variable. = 1;\n",
    "    else &response_variable. = 0;\n",
    "    \n",
    "    * Subset to observations with no missing values for outcome variable or predictor variables;\n",
    "    if nmiss(&response_variable., %sysfunc(tranwrd(&predictor_variables.,%str( ),%str(,)))) = 0;\n",
    "    \n",
    "    keep &response_variable. &predictor_variables.;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 1: Train a Model\n",
    "\n",
    "See page 4 of the paper at https://github.com/saspy-bffs/sgf-2020-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC LOGISTIC output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ods output Association=model_association_table(\n",
    "    where=(Label2='c')\n",
    "    keep=Label2 nValue2\n",
    "    rename=(nValue2=original_model_c_statistic)\n",
    ");\n",
    "proc logistic data=example_dataset;\n",
    "    class &class_variables.;\n",
    "    model &outcome. = &predictor_variables.;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 1: Train a Model (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 2: Generate Bootstrap Samples\n",
    "\n",
    "See page 5 of the paper at https://github.com/saspy-bffs/sgf-2020-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC SURVEYSELECT output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc surveyselect\n",
    "        data=example_dataset\n",
    "        out=bootstrap_samples\n",
    "        seed=1354687\n",
    "        method=urs\n",
    "        outhits\n",
    "        rep=500\n",
    "        samprate=1\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 2: Generate Bootstrap Samples (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 3: Train Models in Each Bootstrap\n",
    "\n",
    "See page 6 of the paper at https://github.com/saspy-bffs/sgf-2020-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Empty output should be created in JupyterLab (but a log is still being produced).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* turn off all output;\n",
    "ods graphics off;\n",
    "ods exclude all;\n",
    "ods noresults;\n",
    "\n",
    "* Trains models on all 500 bootstrap samples, capturing the resulting C-statistics;\n",
    "ods output Association=bootstrap_association_table(\n",
    "    where=(Label2='c')\n",
    "    keep=Replicate Label2 nValue2\n",
    "    rename=(nValue2=c_statistic_value)\n",
    ");\n",
    "proc logistic data=bootstrap_samples outmodel=bootstrap_models;\n",
    "    by Replicate;\n",
    "    class &class_variables.;\n",
    "    model &outcome. = &predictor_variables.;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 3: Train Models in Each Bootstrap (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n",
    "\n",
    "**Warning**: This log entry will be particularly long since each of the 500 iterations of PROC LOGISITIC produces its own log entries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 4: Test Bootstrap Models\n",
    "\n",
    "See page 7 of the paper at https://github.com/saspy-bffs/sgf-2020-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Empty output should be created in JupyterLab (but a log is still being produced).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* Score original dataset with bootstrap models;\n",
    "ods output Scorefitstat=bootstrap_scores(\n",
    "    keep=Replicate AUC\n",
    "    rename=(AUC=c_statistic_value)\n",
    ");\n",
    "proc logistic inmodel=bootstrap_models;\n",
    "    score\n",
    "        data=example_dataset\n",
    "        fitstat\n",
    "    ;\n",
    "    by Replicate;\n",
    "run;\n",
    "\n",
    "* turn output back on;\n",
    "ods results;\n",
    "ods select all;\n",
    "ods graphics on;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 4: Test Bootstrap Models (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 5: Estimate Optimism\n",
    "\n",
    "See page 8 of the paper at https://github.com/saspy-bffs/sgf-2020-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc sql;\n",
    "    create table model_optimism as\n",
    "        select\n",
    "            avg(A.c_statistic_value - B.c_statistic_value) as optimism\n",
    "        from\n",
    "            bootstrap_association_table as A\n",
    "            inner join\n",
    "            bootstrap_scores as B\n",
    "            on A.Replicate = B.Replicate\n",
    "    ;\n",
    "quit;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 6: Adjust Performance with Optimism\n",
    "\n",
    "See page 9 of the paper at https://github.com/saspy-bffs/sgf-2020-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC PRINT output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* assemble C-statistic for original model, optimism, and corrected C-statistic into a 1x3 table;\n",
    "data corrected_model_evaluation;\n",
    "    set model_association_table;\n",
    "    set model_optimism;\n",
    "    \n",
    "    corrected_c_statistic = original_model_c_statistic - optimism;\n",
    "\n",
    "    label\n",
    "        original_model_c_statistic = 'Naive C-Statistic'\n",
    "        optimism = 'Optimism'\n",
    "        corrected_c_statistic = 'Optimism-Corrected C-Statistic'\n",
    "    ;\n",
    "    \n",
    "    keep original_model_c_statistic optimism corrected_c_statistic;\n",
    "run;\n",
    "\n",
    "* print final results;\n",
    "proc print\n",
    "        data=corrected_model_evaluation\n",
    "        noobs\n",
    "        label\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Example &mdash; Step 6: Adjust Performance with Optimism (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}