coursera/text-mining-and-analytics/notes/text data understanding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP Task\n",
    "\n",
    "1. **Lexical analysis**: Figure out what the basic meaningful units in a language are and determenie the meaning of each word.\n",
    "2. **Syntactic analysis**: Determine how words are related eith each other in a sentence, thus revealing the syntactic structure of a sentence.\n",
    "3. **Semantic analysis**: Determine the meaning of a sentence.\n",
    "4. **Pragmatic analysis**: Determine meaning in context.\n",
    "5. **Discourse analysis**: Determine relation between sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLP Challenges\n",
    "\n",
    "1. **Word-level ambiguity**: Multiple syntactic categories and senses.\n",
    "2. **Syntactic ambiguity**: Multiple syntactic structures.\n",
    "3. **Anaphora resolution**: Multiple pronunciation.\n",
    "4. **Presupposition**: Required inferences to understand the meaning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Consideration in NLP Implemention\n",
    "\n",
    "* While linguistic knowledge is always useful, today, the most advanced natural language processing techniques tend to rely on heavy use of statistical machine learning techniques with linguistic knowledge only playing a somewhat secondary role.\n",
    "* Only “shallow” analysis of natural language processing can be done for arbitrary text and in a robust manner; “deep” analysis tends not to scale up well or be robust enough for analyzing unrestricted text. In many cases, a significant amount of training data (created by human labeling) must be available in order to achieve reasonable accuracy.\n",
    "\n",
    "![nlp-difficulty-level](images/nlp-difficulty-level.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Representation\n",
    "\n",
    "1. **String**: Collecting every characters.<br>\n",
    "Pros: General uses.<br>\n",
    "Cons: Not allowed semantic analysis.<br>\n",
    "2. **Word**: Create string segmentation into readable words.<br>\n",
    "Pros: Statistical analysis, +POS tags, +Syntactic structures.<br>\n",
    "Cons: Less general, i. e. Chinese words need more sophiscated segmentation.<br>\n",
    "3. **Graph**: Represent entities and relations.<br>\n",
    "Pros: Semantical analysis.<br>\n",
    "Cons: Deep analysis, not general.<br>\n",
    "4. **Logic**: Represent rules.<br>\n",
    "Pros: Inferences.<br>\n",
    "Cons: May need significant computation time.<br>\n",
    "5. **Speech act**: Represent intent of languages.<br>\n",
    "Pros: Represent human knowledge.<br>\n",
    "Cons: Deep analysis and Limited uses.<br>\n",
    "<br>\n",
    "![text representation](images/text-representation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical Language Model\n",
    "\n",
    "A statistical language model is probability distribution over word sequences. It thus gives any sequence of words a potentially different probability. For example, a language model may give the following three word sequences different probabilities:\n",
    "\n",
    "$$\\eqalign{\n",
    "p(\\text{Today is Wednesday}) &= 0.001\\\\\n",
    "p(\\text{Today Wednesday is}) &= 0.0000000001\\\\\n",
    "p(\\text{The equation has a solution}) &= 0.000001\\\\\n",
    "}$$\n",
    "\n",
    "By calculate the probability of each word sequence, we may reveal document context. For example above, we may deduce that it more likely that some word sequences belong in mathematic context. Thus, a language model can be **context dependent**.\n",
    "\n",
    "One advantage of using language model is it can quantify the uncertaities associated with the use of natural language. For examples:\n",
    "\n",
    "* Help speech recognizer to do word correction. For example: Given correct words *John feels* and next word prediction are *happy* or *habit*. Since, *happy* and *habit* may have similar acoustic signal, thus we can suggest that word *happy* have higher probability that *habit*. So, the correct word should be *John feels happy*.\n",
    "* Help document categorization by count most frequence words.\n",
    "* Predict query based on user preferences. For example, know that user interested in *sport news*, we can suggest any words with high probility in the context of sport.\n",
    "\n",
    "### Unigram language model\n",
    "\n",
    "Assume that each word has independent probability as candidate. Thus, the probabiliy of sequence words equal to the product of the probability of each word.\n",
    "\n",
    "Let:\n",
    "\n",
    "$$\\eqalign{\n",
    "V &: \\text{Set of words in the vocabulary}\\\\\n",
    "\\{w_1, ..., w_n\\} &: \\text{Word sequences, where} \\ w_i \\in V\n",
    "}$$\n",
    "\n",
    "We have:<br>\n",
    "\n",
    "$$p(w_1, ..., w_n) = \\prod\\limits_{i=1}^n p(w_i)$$\n",
    "\n",
    "Given a unigram language model \\\\(\\theta\\\\), we have many parameters as the words in the vocabulary, and they satisfy constraint \\\\(\\sum_{w \\in V} p(w)=1\\\\). Such a model essentially specifies a multinomial distribution over all the words. For example, we define \\\\(\\theta\\\\) as a model to describe topic of the document and document \\\\(D\\\\):\n",
    "\n",
    "Let:<br>\n",
    "$$\\eqalign{\n",
    "\\theta_1 &= \\text{text mining}\\\\\n",
    "\\theta_2 &= \\text{health}\n",
    "}$$\n",
    "\n",
    "Each \\\\(\\theta\\\\) trained in the abstact of text mining paper and health paper, given abstract with uniformly distributed length, such that:<br>\n",
    "$$\\eqalign{\n",
    "p(w|\\theta_1) &= [\\{\\text{text}: 0.2\\}, \\{\\text{mining}: 0.1\\}, \\{\\text{association}: 0.01\\}, \\{\\text{clustering}: 0.02\\}, ..., \\{\\text{food}: 0.00001\\}]\\\\\n",
    "p(w|\\theta_2) &= [\\{\\text{food}: 0.25\\}, \\{\\text{nutrition}: 0.1\\}, \\{\\text{healthy}: 0.05\\}, \\{\\text{diet}: 0.02\\}, ..., \\{\\text{text}: 0.00001\\}]\\\\\n",
    "p(w|\\theta_1) &\\neq p(w|\\theta_2)\n",
    "}$$\n",
    "\n",
    "Then, we expect that any new observed document \\\\(D_i\\\\) is belong to text mining or health topic if only if:\n",
    "$$\\eqalign{\n",
    "p(D_i,\\theta_1) > p(D_i.\\theta_2) \\implies D_i &\\in \\text{text mining}\\\\\n",
    "p(D_i,\\theta_1) < p(D_i.\\theta_2) \\implies D_i &\\in \\text{health}\n",
    "}$$\n",
    "\n",
    "If we have more than two \\\\(\\theta\\\\), then it will be little bit difficult to tell which document belong to. One very simple method to solve this called as **Maximum Likehood Estimation** which difine parameter \\\\(\\hat{\\theta}\\\\) as higest likehood to explain about the document:\n",
    "\n",
    "$$\\hat{\\theta} = arg \\ max_{\\theta} \\ p(D|\\theta)$$\n",
    "\n",
    "which in the case of unigram language model, \\\\(\\hat{\\theta}\\\\) tells us about the probibility of each word equal to its relative frequency in \\\\(D\\\\):\n",
    "\n",
    "$$p(w|\\hat{\\theta}) = \\frac{c(w,D)}{|D|}$$\n",
    "\n",
    "where:\n",
    "$$\\eqalign{\n",
    "c(w,D) &= \\text{count of word} \\ w \\ \\text{in} D\\\\\n",
    "|D| &= \\text{length of} \\ D \\ \\text{document} \\ \\text{or total number of words in} \\ D\n",
    "}$$\n",
    "\n",
    "**The challange is create a model \\\\(p(w)\\\\) as general model in broad document topic**, thus we need penalized bias word, such as *the*, *a*, *is*, etc and normalize significant words that describe topic of the document, such as *computer*, *sport*, *goverment*, etc. Such technique called as **background language model** as shown in image below:\n",
    "\n",
    "![background language model](images/background-language-model.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}