notebooks/ibmseti_intro_features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# IBM SETI Tutorial\n",
    "\n",
    "## Get data from Object Store and calculate features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#!pip install --user --upgrade ibmseti\n",
    "#!pip install --user --upgrade ibmos2spark\n",
    "import ibmseti\n",
    "import ibmos2spark\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "credentials_1 = {\n",
    "  'auth_uri':'',\n",
    "  'global_account_auth_uri':'',\n",
    "  'username':'xxx',\n",
    "  'password':\"xxx\",\n",
    "  'auth_url':'https://identity.open.softlayer.com',\n",
    "  'project':'xxx',\n",
    "  'project_id':'xxx',\n",
    "  'region':'dallas',\n",
    "  'user_id':'xx',\n",
    "  'domain_id':'xx',\n",
    "  'domain_name':'xx',\n",
    "  'container':'seti_raw_data',\n",
    "  'tenant_id':'xx'\n",
    "}\n",
    "\n",
    "configuration_name = 'my_dsx_object_storage' \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "bmos = ibmos2spark.bluemix(sc, credentials_1, configuration_name)\n",
    "rdd = sc.pickleFile(bmos.url('seti_raw_data', 'kepler1229b.sigdb.archive-compamps.rdd.dict.pickle'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Extract the Spectrograms from each file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def extract_spectrogram_and_bins(row):\n",
    "\n",
    "    aca = ibmseti.compamp.Compamp(row['raw_data'])\n",
    "    \n",
    "    spect = ibmseti.dsp.compamp_to_spectrogram(aca)  \n",
    "    \n",
    "    #calculates the frequency (fb) and time (tb) bins\n",
    "    fb = ibmseti.dsp.frequency_bins(aca.header())\n",
    "    tb = ibmseti.dsp.time_bins(aca.header())\n",
    "\n",
    "    return (tb, fb, aca.header(), spect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd = rdd.map(extract_spectrogram_and_bins)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Example Feature: \"standard deviation\" \n",
    "\n",
    "The standard deviation of a set of values is used to measure the \"width\" of the distribution of those values about its central value. In our case, it's used as a proxy to estimate if a signal exists in the spectrogram.\n",
    "\n",
    "We calculate a mean standard deviation for each axis. \n",
    "\n",
    "Consider a spectrogram of N frequency bins (typically 6144) and M time bins (typically 129). Stepping through the calculation, we start with the variance \"along the time axis\".\n",
    "\n",
    "`var_t = ibmseti.features.moment( spectrogram, axis=0, moment=2)` \n",
    "\n",
    "The array `var_t` is of length N, where each element is the variance of the M values for each particular frequency bins. \n",
    "\n",
    "Then `np.sqrt(var_t)` takes the square root and leaves an array of size N. We've calculated the standard deviation, the \"width\", of the M values for each frequency bin. Finally, `std_time` is the mean of those N values. \n",
    "\n",
    "We do the same for the other axis, calculating the mean standard deviation \"along the frequency axis\" for each of the time bins, which we call `std_freq`.  \n",
    "\n",
    "\n",
    "### Typical Narrow Band Signal\n",
    "\n",
    "For a typical narrow-band signal with a relatively small amount of drift, the `std_freq` will be relatively large. This is because at each of the M time intervals, the set of the values across all frequencies will contain a large number of small noise values and a large number of large signal values -- thus, a wide distribution of numbers. Along the time-axis, the `std_time` will be relatively small because for most of the frequency bins, there will be just noise in a small range of values just above zero. There will, of course, be a handful of frequency bins with large widths as the signal passes through that frequency. But the vast majority will be noise, resulting in a small `std_time`.\n",
    "\n",
    "### Wideband Pulse\n",
    "\n",
    "Conversely, a signal that is seen across a wide range of frequencies, but only for a short duration, will have a small `std_freq` and a large `std_time`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def extract_mean_std(row):\n",
    "    std_time = np.mean(np.sqrt(ibmseti.features.moment( row[3], axis=0, moment=2)))\n",
    "    std_freq = np.mean(np.sqrt(ibmseti.features.moment( row[3], axis=1, moment=2)))\n",
    "    return (std_time, std_freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd_features = rdd.map(extract_mean_std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.9 ms, sys: 6.85 ms, total: 12.7 ms\n",
      "Wall time: 22.2 s\n"
     ]
    }
   ],
   "source": [
    "%time npfeat = np.array(rdd_features.collect())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAFSCAYAAACt0ZgcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXXV97/H3NwmkQ7kmeKAYNQhHrKdQxPvB1kFNQtWm\nptFePHjG6jHaQwVh0HAt45HUxjpgfbwGlIxYFQRS4+VxE6Ejps9R2wJKhdZjKii1RC4CoYyQmO/5\nY62BTZgJmcmeWb+deb+eZz/svfZea393Fot8+N1WZCaSJEkq06ymC5AkSdL4DGuSJEkFM6xJkiQV\nzLAmSZJUMMOaJElSwQxrkiRJBSsmrEXErIi4ISLW168XRsS3IuJfI+JzETGn6RolSZKmWzFhDTgV\nuKXt9WpgMDOPAu4D3tJIVZIkSQ0qIqxFxALgVcAlbZtfDlxVPx8Clk13XZIkSU0rIqwBFwHvAhIg\nIuYDP8/M7fX7dwCHNVSbJElSYxoPaxHxamBzZt4ExOjmtuejvC+WJEmacUoYtH88sDQiXgX0APsB\nHwQOiIhZdevaAuCnY+0cEYY4SZLUNTJzxwapnYqSbuQeES8D+jNzaURcDlydmZdHxMeA72bmx8fY\nJ0v6DZqYgYEBBgYGmi5Dk+C5626ev+7luetuETHhsNZ4N+hOnAmcHhE/AOYBn2y4HkmSpGlXQjfo\nozLzG8A36uc/Al7UbEWSJEnNKrllTTNAb29v0yVokjx33c3z1708dzNPUWPWJsMxa5IkqVvsaWPW\nJEmSZjzDmiRJUsEMa5IkSQUzrEmSJBXMsCZJklQww5okSVLBDGuSJEkFM6xJkiQVzLAmSZJUMMOa\nJElSwQxrkiRJBTOsSZIkFcywJkmSVDDDmiRJUsEMa5IkSQUzrEmSJBXMsCZJklQww5okSVLBDGuS\nJEkFM6xJkiQVzLAmSZJUMMOaJElSwQxrkiRJBTOsSZIkFcywJkmSVDDDmiRJUsEMa5IkSQUzrEmS\npK7XarVYvHg5ixcvp9VqNV1OR0VmNl3DbomI7PbfIEmSJq/VarFsWR8jI6sB6OlZybp1QyxZsqTh\nyp4oIsjMmNA+3R50DGuSJM1sixcvZ8OGpUBfvWWIRYvWc801VzVZ1pgmE9bsBpUkSSrYnKYLkCRJ\n2h39/SvYuLGPkZHqdU/PSvr7h5otqoPsBpUkSV2v1WoxOLgGqMJbiePVwDFrkiRJRXPMmqQZbU+e\nui9p5rJlTdIeoZum7kuaubqyGzQi5gLXA3tTTXi4MjPfExGXAi8D7gcSeFNmfm+M/Q1rkrpq6r6k\nmWsyYa3x2aCZ+XBEnJCZD0XEbODvI+Jr9dtnZObVTdYnSZLUpMbDGkBmPlQ/nUtV0/b69YSSp6SZ\na0+fui9p5mq8GxQgImYB/wQcAXwkM8+qu0FfDDwMXAucmZlbx9jXblBJQPdM3Zc0c3XlmLV2EbE/\nsA54B3BPZm6OiL2Ai4EfZuYFY+xjWJMkSV2hK8estcvMByLiG8CJmXlhvW1r3crWP95+AwMDjz7v\n7e2lt7d3iiuVJEl6csPDwwwPD+/WMRpvWYuIg4GtmXl/RPQALeAvgRsy886ICOBCYCQzzx5jf1vW\nJElSV+jWlrVfA4bqcWuzgMsz86sRcW0d5AK4CXh7k0VKkiQ1ofGWtd1ly5okSeoW3m5KkiRpD2NY\nkyRJKphhTZIkqWCGNUmSpIIZ1iRJkgpmWJMkSSqYYU2SJKlghjVJkqSCGdYkSZIKZliTJEkqmGFN\nkiSpYIY1SZKkghnWJEmSCmZYkyRJKphhTZIkqWCGNUmSpIIZ1iRJkgpmWJMkSSqYYU2SJKlghjVJ\nkqSCGdYkSZIKZliTJEkqmGFNkiSpYIY1SZKkghnWJEmSCmZYkyRJKphhTZIkqWCGNUmSpIIZ1iRJ\nkgpmWJMkSSqYYU2SJKlghjVJkqSCGdYkSZIKZliTJHWNVqvF4sXLWbx4Oa1Wq+lypGkRmdl0Dbsl\nIrLbf4Mk6cm1Wi2WLetjZGQ1AD09K1m3boglS5Y0XJm06yKCzIyJ7GPLmqQJs3VDTRgcXFMHtT6g\nCm2Dg2uaLkuacnOaLkBSd9mxdWPjxj5bNyRpCjUe1iJiLnA9sDdVPVdm5nsiYiHweeAg4AbgjZm5\nrak6JVUe37oBIyPVNsOaplp//wo2buxjZKR63dOzkv7+oWaLkqZB492gmfkwcEJmPhc4FvidiHgR\nsBoYzMyjgPuAtzRYpiSpYUuWLGHduiEWLVrPokXrbdHVjFHUBIOI2Ieqle1/A18GDs3M7RHxYmAg\nM08cYx8nGEjTyEHekjR5XTvBICJmRcSNwJ3ABmATcF9mbq8/cgdwWFP1SXqMrRuSNL1Ka1nbH1gH\nnA98KjOfVW9fAHwlM39zjH1sWZMkSV1hMi1rjU8waJeZD0TEN4AXAwdGxKy6dW0B8NPx9hsYGHj0\neW9vL729vVNcqSRJ0pMbHh5meHh4t47ReMtaRBwMbM3M+yOiB2gBf0k11ezqzLw8Ij4GfDczPz7G\n/rasSZKkrjCZlrUSwtrRwBDV+LlZwOWZuSoiDuexpTtuBE7KzK1j7G9YkyRJXaErw9ruMqxJkqRu\n0bWzQSVJkjQ2w5okSVLBDGuSJEkFM6xJkiQVzLAmSZJUMMOaJElSwQxrkiRJBTOsSZIkFcywJkmS\nVDDDmiRJUsEMa5IkSQUzrEmSJBXMsCZJklQww5okSVLBDGuSJEkFM6xJkiQVzLAmSZJUMMOaJElS\nwQxr0gzUarVYvHg5ixcvp9VqNV2OJGknIjObrmG3RER2+2+QplOr1WLZsj5GRlYD0NOzknXrhliy\nZEnDlUnSni8iyMyY0D7dHnQMa9LELF68nA0blgJ99ZYhFi1azzXXXNVkWZI0I0wmrNkNKkmSVLA5\nTRcgaXr1969g48Y+Rkaq1z09K+nvH2q2KEnSuOwGlWagVqvF4OAaoApvjleTpOnhmDVJkqSCNTJm\nLSJOi4hVEbF//fpNEfFru3tcSZIkdWaCwb3AKuD1AJm5Fnh1B44rSZI043UirB2QmQ8Bv2jbNtKB\n40qSJM14nQhrcyLiXTsca34HjitJkjTjdSKsXQT8V+ATEfGdiLgemNDAOUmSJI2tY7NBI+LZwDHA\nDzLzpo4cdNe+19mgkiSpKzR6B4PM/BfgGuAVEfGsTh1XkiRpJuvE0h2nR8Q/R8QVwMuBDwEv3e3K\nJEmS1JGWtXlUy3Z8CXgrsBl4XQeOK0mSNON14t6gP8jMW4Fbgcsi4kAev4yHJEmSJqkTLWt3RMRL\nRl9k5n2ZaViTJEnqgE6Etd8F/i4ivhkRAxHxWxHRiRY7SZKkGa8TYe0nwEHAu4BtwHuBaVu6Q5Ik\naU/WibC2JTNHMvNbmXlBZvZm5m/s6s4RsSAirouIWyLi5oh4R739/Ii4IyJuqB8ndqBWSZKkrtKJ\nsPbdiPij3dh/G3B6Zj4HeAnwZ/UCuwAXZuZx9eNru12ppGK0Wi0WL17O4sXLabVaTZcjScXqxNiy\ns4GjImI1cB1wLbAhMzfvys6ZeSdwZ/38wYi4FXhq/ba3rZL2QK1Wi2XL+hgZWQ3Axo19rFs3xJIl\nSxquTJLK04mWteuBo4EXAl8HXgFcOZkDRcRC4Fjg2/WmkyPipoi4JCIO2P1SJZVgcHBNHdT6gCq0\nDQ6uabosSSpSJ8LapcAbgYcy828y808y87cmepCI2Jcq5J2amQ8CHwWOyMxjqVreLuxArZIkSV1l\nt7tBM/PnVIGNesmOzMxfTuQY9X5XApdl5hfr497V9pGLqe6QMKaBgYFHn/f29tLb2zuRr5c0zfr7\nV7BxYx8jI9Xrnp6V9PcPNVuUJE2B4eFhhoeHd+sYkZkT3yniZuDVmfnjiHgesBD4MlVL3VuB72fm\ntRM43qeBuzPz9LZth9bj2YiI04AXZOYbxtg3J/MbJDWr1Wo92vXZ37/C8WqSZoSIIDMnNCZ/smHt\nWKpAtjUiLgb2Ak4ArgbWAi/KzF0agBIRx1ONe7sZyPpxNvAGqvFr24HbgLeNNWnBsCZJkrrFtIW1\nHb70ZOAKYAQ4qX7clJl/tlsH3vXvN6xJkqSu0FRYC2ApcG09MWBaGdYkSVK3mExY68Q6a8ePTgqI\niPnAr2Tmv3fguJIkSTPepJbuiIj/HhH/KyKeBvx221v3Aq+IiFd2pDpJRfBuA5LUnMlOMDgSWAks\nogp8X6G6e8FwZt4VEW/KzLWdLHQntdgNKk2hHe820NOz0rsNSNIkNTVm7cPAPwAvo2plewS4MTP/\nx24deNe/37AmTaHFi5ezYcNSqrsNAAyxaNF6rrnmqibLkqSuNKVj1iLi90bHpu3ga5n5ZWCo/tw8\n4L6JFCFJkqSx7VJYi4i9gasj4vmZeWP7e3VQa399bwfrk9Qw7zYgSc3apW7QiJhLtY7aizPzO1Ne\n1QTYDSpNPe82IEmdMWVj1trC2oep7iowH/gh8DeZecUkau0Yw5okSeoWkwlrE126483ArcBXgb2B\noYi4rh6nJkmSpA6baMvan2bmJ9q2PwU4D3ge8NImmrhsWZMkSd1iylrWMvNh4GGqrs/27Xdl5inA\nVTw2r18qjou6SpK61US6QTcCY96ZIDMvBF7YkYqkDhtd1HXDhqVs2LCUZcv6DGySpK4xkbB2LnBq\nRLxunPe3d6AeqeMGB9fUq+/3AdVK/KMzGyVJKt0uL4qbmd+OiLcAl0XECuAzwE3ANuAtwG1TUqG0\nm+6+ezPwcWA9sKLhaiRJmphdDmsAmfm5iLgFeD/wKWB0gNxngXd3uDZpt7VaLb7//R8Af1VvOYm9\n995Gf//nmyxLkqRdNul7g9YzQZ8J/Htm3tHRqiZWh7NBNa6x7mv53Odeyg03DDdYlSRppprSe4Pu\nKDPvAu6a7P5SUw4+eH7TJUiStMsmHdakbuB9LSVJ3W7S3aClsBtUT8b7WkqSSjFl9wYtmWFNkiR1\ni+m4N6hUNO9UIEna09iypj3G6J0KqgVwq/Fp69YN2e0pSSqG3aCa0cZapmPRovVcc81VTZYlSdKj\n7AaVJEnaw7h0h/YYLtMhSdoT2Q2qPYrLdEiSSuaYNUmSpII5Zk2SJGkPY1iTJEkqmGFNkiSpYIY1\nSZKkghnWJEmSCmZYkyRJKphhTdPOm61LkrTrXGdN08qbrUuSZjIXxVXxvNm6JGkm68pFcSNiQURc\nFxG3RMTNEXFKvf2giLgmIv41IloRcUDTtUqSJE23xsMasA04PTOfA7wEODking2cCXw9M48CrgPO\narBGdUh//wp6elYCQ8BQfbP1FU2XJUlSsYrrBo2IvwU+XD9elpmbI+JQYDgznz3G5+0G7TLebF2S\nNFN1/Zi1iFgIDAO/AfwkMw9qe++ezJw/xj6GNUmS1BW6cszaqIjYF7gSODUzHwRMYJIkacab03QB\nABExhyqoXZaZX6w3b46IQ9q6QX823v4DAwOPPu/t7aW3t3cKq5UkSdo1w8PDDA8P79YxiugGjYhP\nA3dn5ult21YD92bm6ohYCRyUmWeOsa/doAVwHJokSU+uK8esRcTxwPXAzVRdnwmcDXwHuAJ4GvBj\n4PWZed8Y+xvWGuZCt5Ik7ZquDGu7y7DWPBe6lSRp13T1BANJkiQ9URETDNTd+vtXsHFjHyMj1etq\noduhZouSJGkPYTeoOsIJBpIkPTnHrEmSJBXMMWsz1KpVq5g//0jmzz+SVatWNV2OJEnqIFvWukyr\n1eKss97L7bffyTOesYBjjlnI0NA64EP1J06hr28Za9eubbBKSZI0FrtB93CtVovXvGY527bNBZ4F\nHA98CriI9mUz4DS+9rXPOW5MkqTC2A26hzvrrPeybdtewIXA24HPAE8d45NHPTrYX5IkdTeX7ugi\nt99+J/BBHmtFAzgNOKXt9UrgJOBH01iZJEmaKrasFWZnkwUOOmi/MfY4Cngr8E7gA8BJ9PR8hv7+\nFVNfrCRJmnK2rBVk1apVnHvu+xmdLHDuuVWL2TnnnEOr1eInP7mNKpSNOoOqK3QJcDTz5r2X5z3v\nR/T3e19OSZL2FE4wKMj++z+DLVv+D1U3ZwsYYPbsTXzlK5cxOLimvv/mxcAvgbuA11K1psGO9+N0\nkVpJksrjbNAut9deh7Bt2/uBQ6kC22oAIt7Jr/5qDw8++L62906imvk5GtZOZd999+HMM0/m+c9/\nPsuW9TEyUu3f07OSdetsbZMkqWmGtS7QarU4+eTTuf32u+np+RVWrlzBOeecA8CRR/43Nm26A3gO\n1WzP9uU4zgPup+oivZlZs9ZyyCEHcNdd97Bt2yxgHlVL28UcccQCNm169+P2b291kyRJzZhMWHPM\n2jR6bJ202cCH2LKlGpc2PDzMDTf8iK1bHyHiYTJvHWPvLcAi4FzgPznwwDl1UNuLx1rX3gW8ldtu\nWzstv0eSJE09w9o0Ghxcw7ZtBwEX8Fir1818/esX034HAthKNXngZuArwM+A7VTj2BYCf8i9934K\n+HUe3wIH8HFgFj09KxkZqbb09Kykv39o6n6YJEmaMoa1xv09VVBrD1znAi+iCl49VIvg3gx8kqor\n9GPA/HGO9wMWLnwqH/nIX7VNMHC8miRJ3coxa9Nox27Qyjt5/EK3ZwCXUrWkHQocCMwFvlt/DqqF\ncB8CZgP70D7JYNasrXz1q1cbziRJKpATDLpA+wSDvfaaTeYWfvGLWVTh7WKqFrQPUYW2bTwW0NrX\nVBuian27jyOOOJyf/ex+IDnyyKfzvvedZ1CTJKlQhrUuUi2AewGwN9UdCD5M1SvdUz8eAgZ5/IzQ\n9cBVjC7Zsd9+D/DAA7dPd+mSJGmSnA3aJVqtFued9wFgX+BIqgA2myqkjXZpvnOMPX9KFdROBbay\ncuXZ01CtJElqki1rDTjuuF5uvPH5wCXAXwNnUs3ybJ/ZeUbb+1CNU9sKzGbOnF8yMHDmo+uzSZKk\n7mDLWhdYtWoVN974PeBhqiDWB7x7jE8eTXVbqY/Xrx8GttLX9wbWrl07HaVKkqQCGNamUavV4s//\nfBB4M/CptncOBr5H1Zo26hSqgPYv9euHueCC99iaJknSDGNYm0aDg2vYvv0iqta0u6kCGcDvUE0w\n2ErV3ZnAI8BW5s2bxT333NNEuZIkqQCzmi5g5lpLdfuo0bFpALOYPTu44IIzyBwhMw1qkiTNcLas\nTaP+/hVs3Nj36G2gYAM9PXM555x32b0pSZLG5GzQadZqtdpuA7XCBWwlSZpBXBRXkiSpYJMJa45Z\nkyRJKphhTZIkqWCGNUmSpIIZ1iRJkgpmWJMkSSqYYU2SJKlghjVJkqSCGdYkSZIKZlgrTKvV4sgj\nn8teex3CfvsdxqpVq5ouSZIkNajxsBYRn4yIzRHxvbZt50fEHRFxQ/04sckap8OqVavYZ5/DOPHE\n5WzatI1t257Jgw/+J+ee+xcGNkmSZrDGbzcVES8FHgQ+nZnH1NvOB7Zk5oW7sH9X326q1Wpx8sln\nsmnT/wMOBX4O/AlwNHAGcDDz5m3lnnt+2GSZkiSpA7rydlOZuZEqoexoQj+k27RaLY477qW86lV/\nzKZNrwB6gPOAC4HPUAW3D1DlWEmSNFPNabqAnTg5It4I/CPQn5n3N11Qp7RaLZYt62Nk5HDgImA9\nVTDra/vUGmApsIXTT397A1VKkqQSNN6yNo6PAkdk5rHAnVTNTXuMwcE1jIysBg7byad+CpxCX99r\nOeecc6apMkmSVJoiW9Yy8662lxcDX9rZ5wcGBh593tvbS29v75TU1XkrqFrTTqIanzbqVI444ul8\n5CNXsGTJkmZKkyRJu214eJjh4eHdOkbjEwwAImIh8KXMPLp+fWhm3lk/Pw14QWa+YZx9u26CwWPd\noKuBm5k1ay2HH34Y++8/j4MPnk9//wpDmiRJe6DJTDBoPKxFxGeBXmA+sBk4HzgBOBbYDtwGvC0z\nN4+zf9eFNagC2+DgGgDDmSRJM0RXhrXd1a1hTZIkzTxduXSHJEmSxmdYkyRJKphhTZIkqWCGNUmS\npIIZ1iRJkgpmWJMkSSqYYU2SJKlghjVJkqSCGdYkSZIKZliTJEkqmGFNkiSpYIY1SZKkghnWJEmS\nCmZYkyRJKphhTZIkqWCGNUmSpIIZ1iRJkgpmWJMkSSqYYU2SJKlghjVJkqSCGdYkSZIKZliTJEkq\nmGFNkiSpYIY1SZKkghnWJEmSCmZYkyRJKphhTZIkqWCGNUmSpIIZ1iRJkgpmWJMkSSqYYU2SJKlg\nhjVJkqSCGdYkSZIKZliTJEkqmGFNkiSpYIY1SZKkghnWJEmSCtZ4WIuIT0bE5oj4Xtu2gyLimoj4\n14hoRcQBTdYoSZLUlMbDGnApsGSHbWcCX8/Mo4DrgLOmvSpNi+Hh4aZL0CR57rqb5697ee5mnsbD\nWmZuBH6+w+bfA4bq50PAa6e1KE0b/6PTvTx33c3z1708dzNP42FtHP8lMzcDZOadwFMarkeSJKkR\npYY1SZIkAZGZTddARDwD+FJmHlO/vhXozczNEXEo8HeZ+evj7Nv8D5AkSdpFmRkT+fycqSpkgqJ+\njFoPvAlYDfQBXxxvx4n+YEmSpG7SeMtaRHwW6AXmA5uB84G/Bb4APA34MfD6zLyvqRolSZKa0nhY\nkyRJ0vi6aoLBOAvonh8Rd0TEDfXjxCZr1NgiYkFEXBcRt0TEzRFxSr3dBZC7wBjn7x31dq+/wkXE\n3Ij4dkTcWJ+78+vtCyPiW/W197mIKGVYjNrs5PxdGhH/Vm+/ISKOabpWjS0iZtXnaH39esLXXle1\nrEXES4EHgU+3TUY4H9iSmRc2Wpx2qp4ocmhm3hQR+wL/RLWe3p8A92Tm+yNiJXBQZp7ZZK16op2c\nvz/E6694EbFPZj4UEbOBvwdOBU4HrszML0TEx4CbMvMTjRaqMY1z/t5ONTHv6mar05OJiNOA5wH7\nZ+bSiLicCV57XdWyNs4CuvD4yQkqUGbemZk31c8fBG4FFuACyF1hnPP31Pptr7/CZeZD9dO5VBPL\nEjgBuKrePgQsa6A07YIxzt/2+rXXXuEiYgHwKuCSts0vZ4LXXleFtZ04OSJuiohL7EYrX0QsBI4F\nvgUc4gLI3aXt/H273uT1V7i6G+ZG4E5gA7AJuC8zR//SvwM4rKn6tHM7nr/M/If6rQvqa28wIvZq\nsESN7yLgXVT/g0REzAd+PtFrb08Iax8FjsjMY6n+RbY7pmB1F9qVwKl1C0339MNrrPPn9dcFMnN7\nZj6XqjX7hcBY61Z6LRZqx/MXEc8BzqzXH30B1WoKK5usUU8UEa8GNte9EqOtoDsuVQa7cO11fVjL\nzLvysYF3F1P9i6sC1YMorwQuy8zRtfM2R8Qh9fuHAj9rqj7t3Fjnz+uvu2TmA8A3gBcDB0bE6N8B\nC4CfNlaYdknb+TuxrUdiK3ApVQhXWY4HlkbEvwGfo+r+/CBwwESvvW4Ma49LpfVf8KN+H/jnaa9I\nu+pTwC2Z+ddt20YXQIYnWQBZjXvC+fP6K19EHDzaPR0RPcArgVuAvwNeX3/Ma69Q45y/fxm99iIi\nqMb6eu0VJjPPzsynZ+YzgT8CrsvMk5jEtddts0HHWkD3BKrxM9uB24C3jf4fh8oREccD1wM3UzX5\nJnA28B3gClwAuWg7OX9vwOuvaBFxNNUg5ln14/LMXBURhwOfBw4CbgROqltpVJCdnL9rgYOpGi9u\nAt7eNhFBhYmIlwH99WzQCV97XRXWJEmSZppu7AaVJEmaMQxrkiRJBTOsSZIkFcywJkmSVDDDmiRJ\nUsEMa5IkSQUzrEmSJBXMsCZJklSwOU0XIEnjiYi5O2zKzHykkWIkqSG2rEkqSkS8KCI2RMQI8IRH\nRPxeowVK0jQzrEkqRkT8MdVNjv8ReA3wp8BDwIeBFwPPz0xvOC5pRvHeoJKKEBG/CXwLeGNmXtm2\n/RPAr2fmbzdWnCQ1yDFrkkrxfuCb7UGt9gAwf7ydImI28AHgl8CWzHzP1JUoSdPPsCapcRHxFOAV\nwJvHePs3gB/uZPfXAQcDnwCcfCBpj2NYk1SCZwIB3NS+sQ5xvUDfTvZdAlyXmRunrDpJapBhTVIJ\n/r3+57Ydtp8H/N/MvGLHHSJiX+BU4LXAXRHxJqAF/AGwHPgP4AfAycCvUbW6nVZ/x/3AvMy8qD7W\nHGAA+DGwH/CrwNeAFwDHAR/NzH+KiEOBK0bHz0VE7HhMYC1VC+FLgfcBxwD7A4dl5hk7/IZ3ALOB\nLVQTvj5JNalix++8PDNftkt/kpL2OM4GldS4zLwD+CzwltFtEXE68Dyqbs6x9nmQaqzarMxcmZlr\nqYLRh6gC11WZeR7w3Mx8mKqbdHZmfgj4AlXX6aiLgc2ZuaZ+byXwdODzwFzg8Ppzr+SxYMkYx3wK\nVXj8a+DZwDMz85L6+G9rrz8iPg7MzcwPAncBJ9X7fm6M79y8kz8+SXs4Z4NKKkI9UeA0YCHV/0h+\nBxjKnfxHKiKOAz6SmS9p2zYPuDUzD2nbdhRwI1Vg2g78CvCFzHwgIo4FrgeekpkPR8RrgFOA36fq\nmv0hsDAzRyLiEqqWvk+Od0wggX2A72TmM+rv7wXen5kvrF+/APgysCAzt0bEPvU+j9T77/idN2Tm\nRyf1Byup69kNKqkImflLqpayifhNdhjnRjVR4RtjfO6fM/OyMY5xArCxbn17dP/MfDAi/hC4PjNH\n6vd6gQsi4sCdHTMilgLXtW16PfD5iNgvM7cAvw0MZ+ZWgMx8iGo9Ocb5zgt38mcgaQ9nN6ikbnYs\nTwxrr+TxQQmqsWu/aN9Qj3EDuI9qfBsRcQCwlMfC3tOoZ6JGxLOBvYCfAH80zjFHJ0K8Eri27a0/\noOpSfWv9+qfAf7btNysi/mf98rC27zwK2C8zbxnjt0uaIewGldS1IuKbwOmZ+Q9t264F3pKZt+3w\n2VOpuhjvAXqAr2Tmf0RED9UdEjZQhbP3AAfU3ZOHU00SuJKqS3Q58G3g05l5z06O+U3gdZm5uf7u\n9cDVwPdHa42I9wK3UwW+ucC6zLy3/s6/AL5IFQofycw/6OgfnKSuYliT1JXqmZg/Ag7f2bi2CR7z\nRODdmfm0jLMeAAAAr0lEQVTyThxvd0XENcBnMvPTTdciqTl2g0rqKhHx1Ij4GfBC4KudCmq1F/HE\n8W7TJiKeExF31M+PBw4A/qapeiSVwbAmqds8QtWl+LvAn3fqoPVSIW8Gjo2IZ3XquBN0N3BxPX7t\nd4HfqSdeSJrB7AaVJEkqmC1rkiRJBTOsSZIkFcywJkmSVDDDmiRJUsEMa5IkSQUzrEmSJBXMsCZJ\nklQww5okSVLBDGuSJEkF+/9nIFgmLW6KtgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1293019190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 5))\n",
    "plt.scatter( npfeat[:, 1], npfeat[:, 0])\n",
    "plt.xlabel(r'$\\sigma_{frequency}$', fontsize=20)\n",
    "plt.ylabel(r'$\\sigma_{time}$', fontsize=20)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Features In Context\n",
    "\n",
    "Some work has been done already to extract the 'mean_std' features for all archive-compamp files.\n",
    "\n",
    "http://imgur.com/siffvT8\n",
    "\n",
    "From that link, you'll see these data fall right in the middle of that full archive-compamp set. The idea is to come up with features that allow for classification of signals. This is very much unexplored data!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## More features to be considered:\n",
    "\n",
    "* std of spectrogram projected on time and freq axis\n",
    "* third and fourth moments (std is sqrt(2nd moment))\n",
    "* Log(pixel) histogram fits\n",
    "* Entropy\n",
    "* Total Variation\n",
    "* Maximum Variation\n",
    "* Mean first-difference\n",
    "* 2nd, 3rd and 4th moments of first-difference\n",
    "* Excess Kurtosis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}