perform_linear_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"toc\"></a>\n",
    "# PERFORM LINEAR REGRESSION\n",
    "\n",
    "This notebook works through the process of performing linear regression on the different combinations of metrics, properties, and measures.\n",
    "\n",
    "---\n",
    "- [WORKSPACE VARIABLES](#workspace-variables)\n",
    "- [UTILITY FUNCTIONS](#utility-functions)\n",
    "- [PREPARE DATA](#prepare-data)\n",
    "- [RUN REGRESSION](#run-regression)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform linear regression on four different cases that vary in the features and responses:\n",
    "\n",
    "> **Case 1.** graph measures $\\rightarrow$ hemodynamic properties <br />\n",
    "> **Case 2.** hemodynamic properties $\\rightarrow$ emergent metrics <br /> \n",
    "> **Case 3.** graph measures $\\rightarrow$ emergent metrics <br />\n",
    "> **Case 4.** graph measures + hemodynamic properties $\\rightarrow$ emergent metrics\n",
    "\n",
    "The final output of the regression is a single file `LINEAR_REGRESSION.csv` that is used as input to D3 for plotting the regression bar plots ([go to figure](http://0.0.0.0:8000/figures/linear_regression.html))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"workspace-variables\"></a>\n",
    "\n",
    "### WORKSPACE VARIABLES \n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Set up workspace variables for linear regression.\n",
    "\n",
    "- **`ANALYSIS_PATH`** is the path for analysis files (`.json` and `.csv` files, `.tar.xz` compressed archives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANALYSIS_PATH = \"/path/to/analysis/files/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **`NAMES`** is the list of simulation sets to use\n",
    "- **`CONTEXTS`** is the list of contexts (colony and tissue)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAMES = [\"EXACT_HEMODYNAMICS\", \"VASCULAR_FUNCTION\"]\n",
    "CONTEXTS = [\"C\", \"CHX\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **`METRICS`** is the list of emergent metrics and center concentrations\n",
    "- **`PROPERTIES`** is the list of hemodynamic properties\n",
    "- **`MEASURES`** is the list of graph measures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "METRICS = [\"GROWTH\", \"CYCLES\", \"SYMMETRY\", \"ACTIVITY\", \"GLUCOSE\", \"OXYGEN\"]\n",
    "PROPERTIES = [\"PRESSURE\", \"RADIUS\", \"WALL\", \"SHEAR\", \"CIRCUM\", \"FLOW\"]\n",
    "MEASURES = [\"SHORTPATH\", \"GDIAMETER\", \"GRADIUS\", \"ECCENTRICITY\", \"CLOSENESS\", \"BETWEENNESS\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"utility-functions\"></a>\n",
    "\n",
    "### UTILITY FUNCTIONS\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "General utility functions for data preparation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_json_with_time(jsn, metric, context):\n",
    "    \"\"\"Filter json by time and context.\"\"\"\n",
    "    data = [d for d in jsn if d['time'] == 15.0 and d[\"context\"] == context and d['graphs'] != \"PATTERN\"]\n",
    "    assert(len(data) == 5)\n",
    "    return data\n",
    "\n",
    "def filter_json_without_time(jsn, context):\n",
    "    \"\"\"Filter json by context.\"\"\"\n",
    "    data = [d for d in jsn if d[\"context\"] == context and d['graphs'] != \"PATTERN\"]\n",
    "    assert(len(data) == 5)\n",
    "    return data\n",
    "\n",
    "def filter_csv(content, header, measure, context):\n",
    "    \"\"\"Filter csv by context.\"\"\"\n",
    "    data = []\n",
    "    \n",
    "    for layout in ['Lav', 'Lava', 'Lvav', 'Sav', 'Savav']:\n",
    "        d = [float(d[header.index(measure.lower())]) for d in content\n",
    "            if d[header.index(\"context\")] == context\n",
    "            and d[header.index(\"graph\")] == layout]\n",
    "        data = data + d\n",
    "\n",
    "    assert(len(data) == 50)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"prepare-data\"></a>\n",
    "\n",
    "### PREPARE DATA\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Extract metrics, properties, and measures into a single dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to extract some additional properties from the `EXACT_HEMODYNAMICS` and `VASCULAR_FUNCTION` simulation sets.\n",
    "The function `merge_graph` extracts individual properties from the `.GRAPH` files produced in the basic analysis step.\n",
    "Values across conditions and times are merged into individual files for each property (`EXACT_HEMODYNAMICS.GRAPH.*.json`, `VASCULAR_FUNCTION.GRAPH.*.json`).\n",
    "\n",
    "Note that these files are provided, so this block can be skipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.generate import merge_graph, save_graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.EXACT_HEMODYNAMICS import EXACT_HEMODYNAMICS\n",
    "for prop in PROPERTIES:\n",
    "    EXACT_HEMODYNAMICS.loop(ANALYSIS_PATH, merge_graph, save_graph, f\".GRAPH.{prop}\", timepoints=[\"150\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from scripts.VASCULAR_FUNCTION import VASCULAR_FUNCTION\n",
    "for prop in PROPERTIES:\n",
    "    VASCULAR_FUNCTION.loop(ANALYSIS_PATH, merge_graph, save_graph, f\".GRAPH.{prop}\", timepoints=[\"150\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can iterate through the simulation sets and contexts to combine all the data into a single dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scripts.utilities import load_json, load_csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data(path, name, context, responses, properties, measures):\n",
    "    df = pd.DataFrame()\n",
    "    \n",
    "    # add response data by loading from .SEEDS and .CENTERS files\n",
    "    for response in responses:\n",
    "        if response in [\"GLUCOSE\", \"OXYGEN\"]:\n",
    "            D = load_json(f\"{path}{name}/{name}.CENTERS.json\")\n",
    "            d = filter_json_without_time(D['data'], context)\n",
    "            df[response] = [e for entry in d for e in entry[response.lower()]]\n",
    "        else:\n",
    "            D = load_json(f\"{path}{name}/{name}.SEEDS.{response}.json\")\n",
    "            d = filter_json_with_time(D['data'], response, context)\n",
    "            df[response] = [e for entry in d for e in entry[\"_\"]]\n",
    "    \n",
    "    # add property data by loading from .GRAPH files\n",
    "    for prop in properties:\n",
    "        D = load_json(f\"{path}{name}/{name}.GRAPH.{prop}.json\")\n",
    "        d = filter_json_without_time(D, context)\n",
    "        df[prop] = [e for entry in d for e in entry[\"_\"][\"mean\"]]\n",
    "\n",
    "    # add measure data by loading from .MEASURES files\n",
    "    D = load_csv(f\"{path}_/GRAPH_MEASURES.csv\")\n",
    "    header = D[0]\n",
    "    content = D[1:]\n",
    "    context_code = \"C/CH\" if name == \"EXACT_HEMODYNAMICS\" else context.replace(\"CHX\", \"CH\")\n",
    "    for measure in measures:\n",
    "        df[measure] = filter_csv(D, header, measure, context_code)\n",
    "        \n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_df = {}\n",
    "for context in CONTEXTS:\n",
    "    for name in NAMES:\n",
    "        all_df[f\"{context}_{name}\"] = load_data(ANALYSIS_PATH, name, context, METRICS, PROPERTIES, MEASURES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"run-regression\"></a>\n",
    "\n",
    "### RUN REGRESSION\n",
    "<span style=\"float:right;\">[back to top](#toc)</span>\n",
    "\n",
    "Run linear regression of each of four combinations of metrics, properties, and measures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.formula.api import ols\n",
    "from scripts.utilities import save_csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_regression(df):\n",
    "    out = []\n",
    "    \n",
    "    # z-score data frame\n",
    "    ndf = (df - df.mean())/df.std()\n",
    "    \n",
    "    # run regression case 1 (measures -> properties)\n",
    "    for prop in PROPERTIES:\n",
    "        reg = ols(prop + ' ~ ' + \" + \".join(MEASURES), data=ndf).fit()\n",
    "        out.append([1, prop, reg.rsquared, reg.rsquared_adj])\n",
    "        \n",
    "    # run regression case 2 (properties -> metrics)\n",
    "    for metric in METRICS:\n",
    "        reg = ols(metric + ' ~ ' + \" + \".join(PROPERTIES), data=ndf).fit()\n",
    "        out.append([2, metric, reg.rsquared, reg.rsquared_adj])\n",
    "    \n",
    "    # run regression case 3 (measures -> metrics)\n",
    "    for metric in METRICS:\n",
    "        reg = ols(metric + ' ~ ' + \" + \".join(MEASURES), data=ndf).fit()\n",
    "        out.append([3, metric, reg.rsquared, reg.rsquared_adj])\n",
    "    \n",
    "    # run regression case 4 (measures + properties -> metrics)\n",
    "    for metric in METRICS:\n",
    "        reg = ols(metric + ' ~ ' + \" + \".join(MEASURES + PROPERTIES), data=ndf).fit()\n",
    "        out.append([4, metric, reg.rsquared, reg.rsquared_adj])\n",
    "    \n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out = []\n",
    "for context in CONTEXTS:\n",
    "    for name in NAMES:\n",
    "        reg = run_regression(all_df[f\"{context}_{name}\"])\n",
    "        out = out + [[name, context] + entry for entry in reg]\n",
    "\n",
    "header = \",\".join([\"name\", \"context\", \"case\", \"response\", \"r2\", \"r2adj\"]) + \"\\n\"\n",
    "save_csv(f\"{ANALYSIS_PATH}_/LINEAR_REGRESSION\", header, zip(*out), \"\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}