example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scraping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SoSEn Command Line Interface\r\n",
      "Usage: __main__.py scrape [OPTIONS]\r\n",
      "\r\n",
      "  run kg\r\n",
      "\r\n",
      "Options:\r\n",
      "  Input: [mutually_exclusive, required]\r\n",
      "    -q, --queries PATH            Path to a list of queries to use with the\r\n",
      "                                  zenodo API\r\n",
      "\r\n",
      "    -a, --all                     Run a blank search and get all inputs\r\n",
      "    -z, --zenodo_in PATH          Path to a Zenodo Cache .json file. The\r\n",
      "                                  program will not make calls to Zenodo and\r\n",
      "                                  instead use this\r\n",
      "\r\n",
      "  -g, --graph_out PATH            Path to the output knowledge graph file\r\n",
      "  -t, --threshold FLOAT           Threshold for SoMEF\r\n",
      "  -f, --format [json-ld|turtle|nt]\r\n",
      "                                  The output format of the knowledge graph\r\n",
      "  -d, --data_dict PATH            The path to a dictionary that will be used\r\n",
      "                                  both to load and save outputs from SoMEF\r\n",
      "\r\n",
      "  -c, --zenodo_cache PATH         Path to a .json file which will be used to\r\n",
      "                                  save data from Zenodo\r\n",
      "\r\n",
      "  -h, --help                      Show this message and exit.\r\n"
     ]
    }
   ],
   "source": [
    "!python -m sosen scrape -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python -m sosen scrape --all \\\n",
    "    --graph_out zenodo_9.ttl \\\n",
    "    --threshold 0.9 \\\n",
    "    --format turtle \\\n",
    "    --data_dict zenodo_9_data_dict.json \\\n",
    "    --zenodo_cache zenodo_9_cache.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above command will query zenodo with a blank search, extract GitHub urls from each result, and then use SoMEF to extract metadata from those GitHub urls. The final graph is stored in .ttl in zenodo_9.ttl. Note that the above command could take multiple days to run, due to GitHub rate limiting.\n",
    "\n",
    "Notice `--data_dict` and `--zenodo_cache`. These are two files that SoSEn uses to save data while it runs the process, and can be used to resume the scraping at any point. `--zenodo_cache` stores the results from Zenodo once the scraping of Zenodo is complete, and `--data_dict` stores the outputs of SoMEF. Note, however, that `--zenodo_cache` is written to once, but `--data_dict` is stored to periodically, sort of as a checkpoint. Additionally, before making a call to SoMEF to analyze a repository, SoSEn checks if the analysis was already present in `--data_dict`. This means that `--data_dict` is also an input.\n",
    "\n",
    "Next, we will show the command that can be used to resume the scraping, if the previous long-running process fails for some reason. Notice that the command is virtually the same, except instead of the `--all` option, we pass in `zenodo_9_cache.json` file with the `--zenodo_in` option. This skips the Zenodo scraping step and instead uses the data already scraped. Additionally, `zenodo_9_data_dict.json` will contain the metadata that was extracted through SoMEF, and the process will continue to add to it until all records from Zenodo have been examined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python -m sosen scrape \\\n",
    "    --zenodo_in zenodo_9_cache.json \\\n",
    "    --graph_out zenodo_9.ttl \\\n",
    "    --threshold 0.9 \\\n",
    "    --format turtle \\\n",
    "    --data_dict zenodo_9_data_dict.json \\"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Searching the Knowledge Graph\n",
    "Currently, there are three methods for searching the Knowledge Graph via exact keyword matching. There are manual keywords from GitHub, and additional keywords that are extracted from the title and description of software objects, queried using the methods keyword, title, and description, respectively. After the `--method` input, everything else is interpreted as part of the search query. The first 20 matches are printed, ordered first by the number of keywords "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SoSEn Command Line Interface\n",
      "['adversarial', 'machine', 'learning']\n",
      "\n",
      "FOUND KEYWORDS:\n",
      "keyword: https://w3id.org/okn/o/i/Keyword/adversarial, idf: 7.402451520818244\n",
      "keyword: https://w3id.org/okn/o/i/Keyword/machine, idf: 3.915076442915036\n",
      "keyword: https://w3id.org/okn/o/i/Keyword/learning, idf: 3.7312270019430285\n",
      "\n",
      "MATCHES:\n",
      "1. https://w3id.org/okn/o/i/Software/soorya19/sparsity-based-defenses\n",
      "2. https://w3id.org/okn/o/i/Software/mdoucet/refl_ml\n",
      "3. https://w3id.org/okn/o/i/Software/JoshuaE1/supervised-classification-SSH-publications\n",
      "4. https://w3id.org/okn/o/i/Software/bbuelens/energy-balance\n",
      "5. https://w3id.org/okn/o/i/Software/cisprague/Astro.IQ\n",
      "6. https://w3id.org/okn/o/i/Software/fqararyah/tensorflow-1\n",
      "7. https://w3id.org/okn/o/i/Software/raamana/confounds\n",
      "8. https://w3id.org/okn/o/i/Software/smcclatchy/machine-learning-python\n",
      "9. https://w3id.org/okn/o/i/Software/indigo-dc/DEEPaaS\n",
      "10. https://w3id.org/okn/o/i/Software/neelsoumya/butterfly_detector\n",
      "11. https://w3id.org/okn/o/i/Software/JeanSunny999/Breast-Cancer-Wisconsin-Diagnostic-DataSet\n",
      "12. https://w3id.org/okn/o/i/Software/Nikeshbajaj/Machine_Learning_From_Scratch\n",
      "13. https://w3id.org/okn/o/i/Software/bnpy/bnpy\n",
      "14. https://w3id.org/okn/o/i/Software/mortezanear/Matlab-codes-for-vaccine-tracking\n",
      "15. https://w3id.org/okn/o/i/Software/gbotto/bats\n",
      "16. https://w3id.org/okn/o/i/Software/BENR0/textory\n",
      "17. https://w3id.org/okn/o/i/Software/CCS-Lab/easyml\n",
      "18. https://w3id.org/okn/o/i/Software/caiostringari/pywavelearn\n",
      "19. https://w3id.org/okn/o/i/Software/machine-learning-scala/mls\n",
      "20. https://w3id.org/okn/o/i/Software/psesh/Effective-Quadratures\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -m sosen search --method description adversarial machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SoSEn Command Line Interface\n",
      "['machine', 'machine-learning', 'learning']\n",
      "\n",
      "FOUND KEYWORDS:\n",
      "keyword: https://w3id.org/okn/o/i/Keyword/machine, idf: 8.60642432514418\n",
      "keyword: https://w3id.org/okn/o/i/Keyword/machine-learning, idf: 4.4580125416518035\n",
      "keyword: https://w3id.org/okn/o/i/Keyword/learning, idf: 8.09559870137819\n",
      "\n",
      "MATCHES:\n",
      "1. https://w3id.org/okn/o/i/Software/bcbi/PredictMD.jl\n",
      "2. https://w3id.org/okn/o/i/Software/smarie/python-azureml-client\n",
      "3. https://w3id.org/okn/o/i/Software/radtorch/radtorch\n",
      "4. https://w3id.org/okn/o/i/Software/neelsoumya/butterfly_detector\n",
      "5. https://w3id.org/okn/o/i/Software/christopher-beckham/weka-pyscript\n",
      "6. https://w3id.org/okn/o/i/Software/iml-wg/HEP-ML-Resources\n",
      "7. https://w3id.org/okn/o/i/Software/kjappelbaum/ml_molsim2020\n",
      "8. https://w3id.org/okn/o/i/Software/rieck/harry\n",
      "9. https://w3id.org/okn/o/i/Software/rieck/sally\n",
      "10. https://w3id.org/okn/o/i/Software/SommerEngineering/blog-shitty-models\n",
      "11. https://w3id.org/okn/o/i/Software/CCS-Lab/easyml\n",
      "12. https://w3id.org/okn/o/i/Software/EducationalTestingService/skll\n",
      "13. https://w3id.org/okn/o/i/Software/SCCAF/sccaf\n",
      "14. https://w3id.org/okn/o/i/Software/YinLiLin/KAML\n",
      "15. https://w3id.org/okn/o/i/Software/ds4dm/nectar\n",
      "16. https://w3id.org/okn/o/i/Software/hls-fpga-machine-learning/hls4ml\n",
      "17. https://w3id.org/okn/o/i/Software/jmbr/diffusion-maps\n",
      "18. https://w3id.org/okn/o/i/Software/jonathan-g/datafsm\n",
      "19. https://w3id.org/okn/o/i/Software/lil-lab/spf\n",
      "20. https://w3id.org/okn/o/i/Software/maxibor/sourcepredict\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -m sosen search --method keyword machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SoSEn Command Line Interface\n",
      "['kgtk']\n",
      "\n",
      "FOUND KEYWORDS:\n",
      "keyword: https://w3id.org/okn/o/i/Keyword/kgtk, idf: 9.70503661381229\n",
      "\n",
      "MATCHES:\n",
      "1. https://w3id.org/okn/o/i/Software/usc-isi-i2/kgtk\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -m sosen search --method title kgtk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Describing a Match\n",
    "Once we get a match, we can inspect it using `sosen describe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SoSEn Command Line Interface\n",
      "DESCRIBE <https://w3id.org/okn/o/i/Software/usc-isi-i2/kgtk>\n",
      "@prefix sd: <https://w3id.org/okn/o/sd#> .\n",
      "@prefix sosen: <http://example.org/sosen#> .\n",
      "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n",
      "\n",
      "<https://w3id.org/okn/o/i/Software/usc-isi-i2/kgtk> a sd:Software ;\n",
      "    sosen:descriptionKeywordCount 60 ;\n",
      "    sosen:hasDescriptionKeyword <https://w3id.org/okn/o/i/Keyword/add>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/additional>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/adds>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/bug>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/clean>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/columns>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/command>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/commands>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/custom>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/docker>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/expand>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/explode>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/export>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/filter>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/fixes>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/graph>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/installation>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/instructions>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/kgtk>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/knowledge>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/li>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/lift>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/new>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/ns>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/options>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/prefixes>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/refines>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/rename>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/stats>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/support>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/toolkit>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/triples>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/ul>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/updates>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/validate>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/version>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/wd> ;\n",
      "    sosen:hasKeyword <https://w3id.org/okn/o/i/Keyword/efficient>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/etl-framework>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/graphs>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/knowledge-graphs>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/rdf>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/triples> ;\n",
      "    sosen:hasTitleKeyword <https://w3id.org/okn/o/i/Keyword/additional>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/bug>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/commands>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/fixes>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/i2>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/isi>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/kgtk>,\n",
      "        <https://w3id.org/okn/o/i/Keyword/usc> ;\n",
      "    sosen:keywordCount 6 ;\n",
      "    sosen:titleKeywordCount 13 ;\n",
      "    sd:author <https://w3id.org/okn/o/i/Person/usc-isi-i2> ;\n",
      "    sd:description \"\"\"<p>This version of KGTK fixes:</p>\n",
      "<ul>\n",
      "<li>Updates installation instructions to add Docker support</li>\n",
      "<li>Updates stats</li>\n",
      "<li>Refines filter command</li>\n",
      "<li>adds expand and explode commands</li>\n",
      "<li>Refines the clean and validate command with additional options</li>\n",
      "<li>Bug fixes in export to WD triples (additional support for custom ns prefixes)</li>\n",
      "<li>new commands: lift, rename columns</li>\n",
      "<li>...</li>\n",
      "</ul>\"\"\",\n",
      "        \"Knowledge Graph Toolkit \" ;\n",
      "    sd:doi \"10.5281/zenodo.3828068\" ;\n",
      "    sd:downloadUrl \"https://github.com/usc-isi-i2/kgtk/releases\"^^xsd:anyURI ;\n",
      "    sd:executionInstructions \"\"\"To list all the available KGTK commands, run:\n",
      "\n",
      "```\n",
      "kgtk -h\n",
      "```\n",
      "\n",
      "To see the arguments of a particular commands, run:\n",
      "\n",
      "```\n",
      "kgtk  -h\n",
      "```\n",
      "\n",
      "An example command that computes instances of the subclasses of two classes:\n",
      "\n",
      "```\n",
      "kgtk instances --transitive --class Q13442814,Q12345678\n",
      "```\n",
      "\n",
      "\"\"\" ;\n",
      "    sd:hasInstallationInstructions \"\"\"0. Our installations will be in a conda environment. If you don't have a conda installed, follow [link](https://docs.conda.io/projects/conda/en/latest/user-guide/install/) to install it.\n",
      "1. Set up your own conda environment:\n",
      "```\n",
      "conda create -n kgtk-env python=3.7\n",
      "conda activate kgtk-env\n",
      "```\n",
      " **Note:** Installing Graph-tool is problematic on python 3.8 and out of a virtual environment. Thus: **the advised installation path is by using a virtual environment.**\n",
      "\n",
      "2. Install (the dev branch at this point): `pip install kgtk`\n",
      "\n",
      "You can test if `kgtk` is installed properly now with: `kgtk -h`.\n",
      "\n",
      "3. Install `graph-tool`: `conda install -c conda-forge graph-tool`. If you don't use conda or run into problems, see these [instructions](https://git.skewed.de/count0/graph-tool/-/wikis/installation-instructions). \n",
      "\n",
      "\"\"\",\n",
      "        \"\"\"```\n",
      "docker pull uscisii2/kgtk\n",
      "```\n",
      "\n",
      "To run KGTK in the command line just type:\n",
      "\n",
      "```\n",
      "docker run -it uscisii2/kgtk /bin/bash\n",
      "```\n",
      "\n",
      "If you want to run KGTK in a Jupyter notebook, then you will have to type:\n",
      "```\n",
      "docker run -it -p 8888:8888 uscisii2/kgtk /bin/bash -c \"jupyter notebook --ip='*' --port=8888 --allow-root --no-browser\"\n",
      "```\n",
      "\n",
      "Note: if you want to load data from your local machine, you will need to [mount a volume](https://docs.docker.com/storage/volumes/).\n",
      "\n",
      "More information about versions and tags is available here: https://hub.docker.com/repository/docker/uscisii2/kgtk\n",
      "\n",
      "See additional examples in [the documentation](https://kgtk.readthedocs.io/en/latest/install/).\n",
      "\n",
      "\"\"\" ;\n",
      "    sd:hasSourceCode <https://w3id.org/okn/o/i/SoftwareSource/usc-isi-i2/kgtk> ;\n",
      "    sd:hasVersion <https://w3id.org/okn/o/i/SoftwareVersion/usc-isi-i2/kgtk/0.1.0>,\n",
      "        <https://w3id.org/okn/o/i/SoftwareVersion/usc-isi-i2/kgtk/0.1.1>,\n",
      "        <https://w3id.org/okn/o/i/SoftwareVersion/usc-isi-i2/kgtk/v0.2.0>,\n",
      "        <https://w3id.org/okn/o/i/SoftwareVersion/usc-isi-i2/kgtk/v0.2.1> ;\n",
      "    sd:identifier \"10.5281/zenodo.3828068\",\n",
      "        \"10.5281/zenodo.3891993\" ;\n",
      "    sd:issueTracker \"https://github.com/usc-isi-i2/kgtk/issues\"^^xsd:anyURI ;\n",
      "    sd:keyword \"efficient\",\n",
      "        \"etl-framework\",\n",
      "        \"graphs\",\n",
      "        \"knowledge-graphs\",\n",
      "        \"rdf\",\n",
      "        \"triples\" ;\n",
      "    sd:license \"https://api.github.com/licenses/mit\"^^xsd:anyURI ;\n",
      "    sd:name \"usc-isi-i2/kgtk\",\n",
      "        \"usc-isi-i2/kgtk: KGTK 0.2.1: Additional commands and bug fixes\" ;\n",
      "    sd:readme \"https://github.com/usc-isi-i2/kgtk/blob/master/README.md\"^^xsd:anyURI ;\n",
      "    sd:referencePublication \"\"\"@article{ilievski2020kgtk,\n",
      "  title={KGTK: A Toolkit for Large Knowledge Graph Manipulation and Analysis},\n",
      "  author={Ilievski, Filip and Garijo, Daniel and Chalupsky, Hans and Divvala, Naren Teja and Yao, Yixiang and Rogers, Craig and Li, Ronpeng and Liu, Jun and Singh, Amandeep and Schwabe, Daniel and Szekely, Pedro},\n",
      "  journal={arXiv preprint arXiv:2006.00088},\n",
      "  year={2020},\n",
      "  url={https://arxiv.org/abs/2006.00088}\n",
      "}\"\"\",\n",
      "        \"\"\"```\n",
      "@article{ilievski2020kgtk,\n",
      "  title={KGTK: A Toolkit for Large Knowledge Graph Manipulation and Analysis},\n",
      "  author={Ilievski, Filip and Garijo, Daniel and Chalupsky, Hans and Divvala, Naren Teja and Yao, Yixiang and Rogers, Craig and Li, Ronpeng and Liu, Jun and Singh, Amandeep and Schwabe, Daniel and Szekely, Pedro},\n",
      "  journal={arXiv preprint arXiv:2006.00088},\n",
      "  year={2020},\n",
      "  url={https://arxiv.org/abs/2006.00088}\n",
      "}\n",
      "```\n",
      "\"\"\" .\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/aidan/Documents/usc/usc-isi-reu/env/lib/python3.7/site-packages/SPARQLWrapper/Wrapper.py:979: RuntimeWarning: Sending Accept header '*/*' because unexpected returned format 'json' in a 'DESCRIBE' SPARQL query form\n",
      "  warnings.warn(\"Sending Accept header '*/*' because unexpected returned format '%s' in a '%s' SPARQL query form\" % (self.returnFormat, self.queryType), RuntimeWarning)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -m sosen describe https://w3id.org/okn/o/i/Software/usc-isi-i2/kgtk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": [
     "Download the dependencies then configure SoMEF\n",
     "\n",
     "%%bash\n",
     "somef configure\n",
     "\n"
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}