microfim_tutorial_notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "microfim_tutorial_notebook.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wugJz24DOCwc"
      },
      "source": [
        "# Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1AaEWhzPO3j2"
      },
      "source": [
        "Materials avalaible in the repository:\n",
        "\n",
        "*   files for tutorials (dir: **tutorials**)\n",
        "*   files for tests (dir: **test**)\n",
        "*   functions (dir: **functions**)\n",
        "*   template files to be filled (dir: **template_inputs**)\n",
        "*   guided scripts (**script_***)\n",
        "*   requirements.txt (for conda installation)\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ee6isMtEOdEx"
      },
      "source": [
        "Prerequisites:\n",
        "\n",
        "*   Python > 3 (https://www.python.org/)\n",
        "*   Conda or Miniconda (https://conda.io/projects/conda/en/latest/user-guide/install/index.html)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iwh-TgBWFyjP"
      },
      "source": [
        "# cloning repository via git (or download zip folder drictly from the github page)\n",
        "\n",
        "git clone https://github.com/qLSLab/microFim.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fq0KNL4HODxG"
      },
      "source": [
        "# create conda env\n",
        "\n",
        "conda create --name microFIM --file requirements.txt --channel default --channel conda-forge --channel plotly"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFCTei9VOEgX"
      },
      "source": [
        "# Script usage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6KYZdOwqPmy-"
      },
      "source": [
        "Guided scripts must be run in the main directory (within microFIM, after cloning the repository and create the environment). The scripts are 'interactive', with auto-completion for an easy usage.\n",
        "\n",
        "We suggest to create a specific directory for your project, in order to set it for inputs and outputs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zw3X7Ir3ORTs"
      },
      "source": [
        "python script_1_filtertable.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ebx4zXH0zYcl"
      },
      "source": [
        "This script can be used to filter your otu/taxa table based on a list of samples.\n",
        "Files required and mandatory instructions:\n",
        "* otu/esv/taxa table - the column name of OTU or TAXA must be '#ID'\n",
        "* sample list  - the first row of your sample list must be '#SampleID'\n",
        "\n",
        "The script will ask you to set the input directory and the two files mentioned -\n",
        "otu/esv/taxa table and sample list. The format of the file does not matter at this stage,\n",
        "the script will ask you the type of separator.\n",
        "\n",
        "The output file will be a filtered CSV file saved into the input directory\n",
        "(in order to allow subsequent analysis)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ZhbfPmFkdHM"
      },
      "source": [
        "python script_2_tableconversion.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKDDHa37zZB_"
      },
      "source": [
        "This script can be used to convert a otu/esv/taxa table into a list of transactions.\n",
        "At this stage, do not worry about the format of the input. The script will ask\n",
        "you which is the separator.\n",
        "\n",
        "The output will be saved as a list of transactions into input directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U9LGdXqGkcxL"
      },
      "source": [
        "python script_3_microfimcalculation.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cdmH97GYzaQv"
      },
      "source": [
        "This script calculate microbial patterns!\n",
        "Files:\n",
        "- otu/esv/taxa table previously converted in transactions\n",
        "- file with parameters in .csv format (support, zmin and zmax + type of report)\n",
        "    template available in the tutorial folder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mT_shLFpkcze"
      },
      "source": [
        "script_4_additionalmeasures.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E2GOxGHvzZtK"
      },
      "source": [
        "This script calculate additional interest measures that can be used\n",
        "to filter results. Currently, all-confidence metric is available (see README for details)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qwk7rLz8kc17"
      },
      "source": [
        "script_5_generatepatterntable.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cnxfeKoSvYgj"
      },
      "source": [
        "This script can be used to create the pattern table.\n",
        "Inputs:\n",
        "- pattern results;\n",
        "- metadata file;\n",
        "- transactional file.\n",
        "\n",
        "The output will be saved as a CSV dataframe (with and without\n",
        "inrerest measures) into input directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VK6IIV87zT92"
      },
      "source": [
        "# # available from monday 15\n",
        "\n",
        "script_6_generateplots.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0pbypp4gOJze"
      },
      "source": [
        "# Library usage\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W3urtkQtPnQ-"
      },
      "source": [
        "microFIM python functions were divided into thematic sections, in order to promote the integration of new functions and an easy development of the tool. Here we present four scripts commented with the complete frameowrk that can be used on test/test2.csv files and the metadata and parameters related. \n",
        " \n",
        "\n",
        "*   The first one (named microFIM_example_code_1.py) filter the data table and convert it in transactional file. To filter, use a metadata files removing lines of samples you want to exclude;\n",
        "*   The second create calculate patterns and calculate additional measures;\n",
        "*   The third create pattern table;\n",
        "*   The fourth create visualizations.\n",
        "\n",
        "For simplicity, inputs are declared in the first lines."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i1oqpkFgLFXW"
      },
      "source": [
        "### From import to conversion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V2vBxA7LLCh2"
      },
      "source": [
        "Before running:\n",
        "\n",
        "*   specify input files! (first lines)\n",
        "*   pay attention at the last printing messages\n",
        "\n",
        "This script suggests at the end to run a bash script to modify the output file (details after the box script):\n",
        "*   if you have a Linux system run only the 'sed' script\n",
        "*   if you have a Mac, run both 'sed' and 'rm'\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MC_K5av_OQ7k"
      },
      "source": [
        "import os\n",
        "import sys\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import csv\n",
        "from csv import writer\n",
        "import readline\n",
        "import re\n",
        "import string\n",
        "\n",
        "import fim\n",
        "import functions.microdir as md\n",
        "import functions.microfim as mf\n",
        "import functions.microimport as mi\n",
        "import functions.microinterestmeasures as mim\n",
        "\n",
        "\n",
        "\"\"\" microFIM example code on test/test1.csv files\n",
        "of microFIM github repository\n",
        "Input files to run microFIM:\n",
        "- test2.csv\n",
        "- metadata_test2.csv\n",
        "- parameters_test2.csv\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# setting files\n",
        "## dir\n",
        "set_dir = 'test'\n",
        "\n",
        "## metadata\n",
        "metadata = 'metadata_test2.csv'\n",
        "meta_sep = ','\n",
        "\n",
        "## otu/esv/taxa table\n",
        "data_table_name = 'test2.csv'\n",
        "data_sep = ','\n",
        "\n",
        "# SETTINGS\n",
        "## set dir\n",
        "data_dir = md.set_inputs_dir_rev(set_dir)\n",
        "print(data_dir)\n",
        "\n",
        "## SET OUTPUT NAME\n",
        "file_name = mi.output_file_name(data_table_name)\n",
        "\n",
        "# IMPORT FILES\n",
        "## import metadata\n",
        "metadata = mi.import_metadata(metadata, data_dir, meta_sep)\n",
        "print(metadata)\n",
        "\n",
        "## import data table (otu, esv or taxa table)\n",
        "data_table = mi.import_data_table(data_table_name, data_dir, data_sep)\n",
        "print(data_table)\n",
        "\n",
        "\n",
        "# FILTER DATA TABLE VIA SAMPLE METADATA\n",
        "filter_table = mf.filter_data_table(metadata, data_table)\n",
        "print(filter_table)\n",
        "\n",
        "\n",
        "# CONVERT DATA TABLE IN TRANSACTIONAL data\n",
        "\n",
        "t_list = mf.convert_in_transaction_list(filter_table, data_table_name)\n",
        "print(t_list)\n",
        "\n",
        "# save file\n",
        "mf.save_transaction_list(data_dir, t_list, file_name)\n",
        "\n",
        "\n",
        "# TO BE PRINT WHEN RUNNING THIS SCRIPT\n",
        "# remove old output to clean folder\n",
        "output = 'transactions_' + file_name[0]\n",
        "\n",
        "print(f'\\n\\n> File converted and saved as ' + output + '.csv' + ' in ' + data_dir + '\\n\\n')\n",
        "\n",
        "print(f'\\n\\n> Now run from your command line in {data_dir}:\\n\\n \\\n",
        "sed -i -e \"s/,/ /g\" {output}\\n\\n \\\n",
        "rm {output}-e\\n\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EBqaCHztXc4J"
      },
      "source": [
        "# to be run in the output folder\n",
        "# substitute (change {output} with your file name)\n",
        "\n",
        "# Linux\n",
        "sed -i -e \"s/,/ /g\" {output}\n",
        "\n",
        "# Mac\n",
        "rm {output}-e"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U2q5jinVLKQZ"
      },
      "source": [
        "### Pattern extraction and pattern table creation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gWUm79JYLBFR"
      },
      "source": [
        "Before running, specify:\n",
        "\n",
        "\n",
        "*   Type of patterns to be extracted - i, c or m (first lines). \n",
        "    * i = itemsets (standard; see README for details);\n",
        "    * c = closed itemsets;\n",
        "    * m = maximal itemsets.\n",
        "*   Input files (first lines)\n",
        "\n",
        "This script generates as output the pattern list, pattern dataframe, pattern table with and without interest measures.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yNrLw3ga0Idu"
      },
      "source": [
        "import os\n",
        "import sys\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import csv\n",
        "from csv import writer\n",
        "import readline\n",
        "import re\n",
        "import string\n",
        "\n",
        "import fim\n",
        "import functions.microdir as md\n",
        "import functions.microfim as mf\n",
        "import functions.microimport as mi\n",
        "import functions.microinterestmeasures as mim\n",
        "\n",
        "\n",
        "\"\"\" microFIM example code on test/test2.csv files\n",
        "of microFIM github repository\n",
        "Input files to run microFIM:\n",
        "- test2.csv\n",
        "- transactional file 2 (can be obtained with microFIM_example_code_1.py)\n",
        "- metadata_test2.csv\n",
        "- parameters_test2.csv\n",
        "\n",
        "Default is itemsets patterns (i), but also closed (c) and maximal (m) can be calculated.\n",
        "\n",
        "Finally, two pattern tables can be generated:\n",
        "- complete pattern table (patterns + interest measures)\n",
        "- clean pattern table\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# set fim options\n",
        "to_calculate = 'i' # default (can be changed in c or m)\n",
        "\n",
        "# setting files\n",
        "## dir\n",
        "set_dir = 'test'\n",
        "\n",
        "## metadata\n",
        "metadata = 'metadata_test2.csv'\n",
        "meta_sep = ','\n",
        "\n",
        "## otu/esv/taxa table\n",
        "data_table_name = 'test2.csv'\n",
        "data_sep = ','\n",
        "\n",
        "## parameter file\n",
        "par_file = 'parameters_test2.csv'\n",
        "\n",
        "# transactional file\n",
        "trans_file = 'transactions_test2'\n",
        "\n",
        "# set output name\n",
        "output_file = 'patterns_test2'\n",
        "add_interest_file = 'addm_patterns_test2'\n",
        "output_pattern_table = 'pattern_table_test2'\n",
        "\n",
        "\n",
        "# SETTINGS\n",
        "## set dir\n",
        "data_dir = md.set_inputs_dir_rev(set_dir)\n",
        "\n",
        "## SET OUTPUT NAME\n",
        "file_name = mi.output_file_name(data_table_name)\n",
        "\n",
        "\n",
        "# import files\n",
        "\n",
        "# IMPORT FILES\n",
        "## import metadata\n",
        "metadata = mi.import_metadata(metadata, data_dir, meta_sep)\n",
        "\n",
        "## import data table (otu, esv or taxa table)\n",
        "data_table = mi.import_data_table(data_table_name, data_dir, data_sep)\n",
        "\n",
        "# import parameters file\n",
        "\n",
        "## import transactions\n",
        "t = mf.read_transaction(os.path.join(data_dir, trans_file))\n",
        "\n",
        "## import file with paramaters\n",
        "minsupp, zmin, zmax= mi.itemsets_parameters(data_dir, par_file)\n",
        "\n",
        "\n",
        "# FILTER DATA TABLE VIA SAMPLE METADATA\n",
        "filter_table = mf.filter_data_table(metadata, data_table)\n",
        "\n",
        "# calculate patterns\n",
        "results = mf.fim_calculation(t, to_calculate, minsupp, zmin, zmax)\n",
        "\n",
        "# write patterns results\n",
        "file, out_file, new_out_file = mf.write_results(results, data_dir, output_file)\n",
        "\n",
        "# convert itemsets results into a dataframe\n",
        "df = mf.itemsets_dataframe(new_out_file)\n",
        "print(df)\n",
        "\n",
        "# export as a csv\n",
        "mf.export_dataframe(df, data_dir, output_file)\n",
        "\n",
        "print('Results saved as ' + new_out_file + ' in ' + data_dir + '\\n\\n')\n",
        "\n",
        "\n",
        "\n",
        "# CALCULATE ADDITIONAL METRICS\n",
        "## import patterns dataframe\n",
        "df = mi.import_pattern_dataframe(data_dir, output_file)\n",
        "\n",
        "# calculate and add all-confidence values\n",
        "data_allc_update = mim.add_interest_measures(data_table, df, trans_file, data_dir)\n",
        "\n",
        "# export dataframe\n",
        "mim.add_table_export(data_allc_update, data_dir, add_interest_file)\n",
        "print('Results saved as df_' + add_interest_file + '.csv in ' + data_dir + '\\n\\n')\n",
        "\n",
        "\n",
        "\n",
        "# GENERATE PATTERN TABLE\n",
        "## generate pattern table with patterns and interest measures\n",
        "pattern_table_complete = mf.generate_pattern_table(data_allc_update, df, data_dir, trans_file, metadata, meta_sep)\n",
        "print(pattern_table_complete)\n",
        "\n",
        "## export pattern table as cv\n",
        "mf.export_pattern_tables(pattern_table_complete, data_dir, output_pattern_table)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nCKBD31ajDGL"
      },
      "source": [
        "## Visualizations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BQD8Z2Bf10pt"
      },
      "source": [
        "To create heatmap based on Jaccard distance, use as input the pattern table. To create bar charts and scatter plots, also pattern dataframe can be used."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t4LpmhCP2SdG"
      },
      "source": [
        "import pandas as pd\n",
        "import plotly.express as px\n",
        "import os\n",
        "import sys\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import readline\n",
        "\n",
        "import functions.microfim as mf\n",
        "import functions.microdir as md\n",
        "import functions.microinterestmeasures as mim\n",
        "import functions.microimport as mi\n",
        "import functions.microplots as mp\n",
        "\n",
        "import plotly.graph_objects as go\n",
        "from plotly.subplots import make_subplots"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J_Q94idB0Is-"
      },
      "source": [
        "# functions for plotting results\n",
        "\n",
        "data_dir = 'test2'\n",
        "meta_sep = ','\n",
        "plot_name = 'heatmap'\n",
        "output_format = 'SVG'\n",
        "\n",
        "\n",
        "df_pattern = mi.import_pattern_dataframe_plot(data_dir, pattern_input)\n",
        "print(df_pattern)\n",
        "\n",
        "metadata = mi.import_metadata(meta_input, data_dir, meta_sep)\n",
        "print(metadata)\n",
        "\n",
        "\n",
        "# heatmap\n",
        "dis_matrix = mp.calculate_distance_matrix(df_pattern, metadata)\n",
        "mp.plot_heatmap(dis_matrix, metadata, data_dir, plot_name, output_format)\n",
        "\n",
        "\n",
        "## EXAMPLES for hist and scatter (2d, 3d)\n",
        "set_col = mp.set_col_name(plot_name)\n",
        "\n",
        "# set columns for plotting\n",
        "cols = ['Pattern length', 'Support', 'All-confidence']\n",
        "\n",
        "# PLOTTING SCATTER 2D\n",
        "col_to_group = 'Support'\n",
        "col_to_group1 = 'Support'\n",
        "col_to_group2 = 'Pattern length'\n",
        "\n",
        "# PLOTTING HIST\n",
        "hist = mp.hist_plot(df_pattern, col_to_group)\n",
        "# # save as SVG and HTML\n",
        "mp.save_hist_plot(outputdir, plot, plot_name, format, set_col)\n",
        "\n",
        "\n",
        "# PLOTTING SCATTER 2D\n",
        "scatter_2d = mp.scatter_plot_2d(df_pattern, col_to_group1, col_to_group2)\n",
        "mp.save_scatter_2d_plot(data_dir, plot, plot_name, format)\n",
        "\n",
        "\n",
        "# PLOTTING SCATTER 3D\n",
        "col_to_group1 = 'Support'\n",
        "col_to_group2 = 'Pattern length'\n",
        "col_to_group3 = 'All-confidence'\n",
        "scatter_3d = mp.scatter_plot_3d(df_pattern, col_to_group1, col_to_group2, col_to_group3)\n",
        "mp.save_scatter_2d_plot(data_dir, plot, plot_name, format)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DZei-UgSWlXu"
      },
      "source": [
        "# Integration in QIIME2 framework"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k-R0CIPvduij"
      },
      "source": [
        "## Export taxa tables for microFIM analysis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4m3DZ4fNWnRB"
      },
      "source": [
        "# activate the env (if you do not installed QIIME2 yet, please see https://docs.qiime2.org/2021.8/getting-started/)\n",
        "\n",
        "conda activate qiime2-2020.8 # example version\n",
        "\n",
        "\n",
        "# export biom file form qza\n",
        "\n",
        "qiime tools export --input-path table.qza --output-path exported-feature-table\n",
        "\n",
        "\n",
        "# convert biom file to tsv\n",
        "\n",
        "biom convert -i exported-feature-table/feature-table.biom -o feature-table.tsv --to-tsv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c0szuTJsGAp1"
      },
      "source": [
        "# substitue #OTU ID with #ID\n",
        "\n",
        "sed -i -e \"s/#OTU ID/#ID/g\" feature-table.tsv\n",
        "\n",
        "\n",
        "# remove first row\n",
        "\n",
        "sed -i '1d' feature-table.tsv\n",
        "\n",
        "\n",
        "## READY TO BE IMPORTED IN microFIM ##"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b0efzf4ddw7R"
      },
      "source": [
        "## Import pattern tables in qza format to perform QIIME2 analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NU0CpEFCUuK7"
      },
      "source": [
        "Change 'Pattern' column in #OTU ID before converting."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dDQ0LV7ydy5U"
      },
      "source": [
        "# convert in biom file\n",
        "\n",
        "biom convert -i pattern_table_test.tsv \\\n",
        "  -o pattern_table_test.biom --table-type=\"OTU table\" --to-json\n",
        "\n",
        "\n",
        "# import in qiime2\n",
        "\n",
        "qiime tools import \\\n",
        "  --input-path pattern_table_test.biom \\\n",
        "  --type 'FeatureTable[Frequency]' \\\n",
        "  --input-format BIOMV100Format \\\n",
        "  --output-path pattern_table_test.qza"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S8DadyrHZ6sX"
      },
      "source": [
        "# Contacts\n",
        "For any doubt, please contact g.agostinetto@campus.unimib.it"
      ]
    }
  ]
}