book/chapters/workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Workflow Management / Cloud Computing\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workflow Management: GeoWeaver\n",
    "\n",
    "The model has been adapted to the open-source workflow management sytem, [GeoWeaver](https://github.com/ESIPFed/Geoweaver). Geoweaver is a web-based application for interactive, full-stack machine leanring workflow management. The app provdies a user-friendly GUI to interact with and persistently store script logs and code revision histories. Shell and python scripts are supported by the platform, as well as seammless intergration wth Jupyter. \n",
    "\n",
    "To install and open a Geoweaver instance perform the following steps;\n",
    "  1. Download the latest release of [geoweaver.jar](https://github.com/ESIPFed/Geoweaver/releases/download/latest/geoweaver.jar).\n",
    "  2. Path to the download folder containing the .jar file\n",
    "  3. Run the command, java -jar geoweaver.jar\n",
    "  4. Open a browser and enter: http://localhost:8070/Geoweaver/ \n",
    "  \n",
    "You will now see the Geoweaver front page. Here you can create, import, edit, and execute processes and build workflows. Please refer to the GEOWEAVER documentation for a full overivew of capabilities. \n",
    "<img align = 'center' src=\"./Images/Geoweaver_home.jpg\" alt = 'drawing' width = '1000'/>\n",
    "\n",
    "The National Snow Model has been adpated to execute in Geoweaver. A full-stack workflow of the model can be found at the github page, [here](https://github.com/AlabamaWaterInstitute/National-Snow-Model/tree/main/tutorials). After downloading the zip folder, import the JSON workflow file to Geoweaver using the \"import wrokflow\" tool in Weaver and the model will be displayed in the weaver window. This workflow is set up to execute from the local machine.\n",
    "<img align = 'center' src=\"./Images/Geoweaver_model.jpg\" alt = 'drawing' width = '1000'/>\n",
    "\n",
    "For simplicity, the National Snow Model workflow is reduced to a single sub-region in Colorado. Each node in the workflow has been assigned to a stage in the model devlopment and exectuition process, from training to interactive SWE mapping. Python scripts of each node may be interacted with under the Processes header. Any edits to processes made here will be relected in the Weaver node and a history of revisions saved in Geoweaver. \n",
    "\n",
    "Training stages of the model can take a long time to execute on a personal computer. The donwloaded model has already undergone trianing, so these steps are not neccessary to perform for the prediction stages of the model. A node may be skipped by double-clicking and selecting \"skip\". Before exectuion it is necessary to update the line below in the Folder_Path process to point to the root folder containing the data and models.\n",
    "     os.chdir(r\"C:\\Users\\My Name\\Desktop\\Geoweaver_ML_SWE\")\n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cloud Computing: Binder\n",
    "<img align = 'center' src=\"./Images/Binder.JPG\" alt = 'drawing' width = '1000'/>\n",
    "\n",
    "Running tutorial workflows can be helpful for new users to toubleshoot during model development.\n",
    "[Binder](https://mybinder.org/) is a free hosting site where users can run their models and data processing workflows for free on the cloud.\n",
    "We have made effort to run the scripts on Binder, however, there are many limitations that challenge the operation of large workflows such as this.\n",
    "For example, there are memory limitations and only 2 CPUs supported on Binder.\n",
    "Further more, Binder uses python 3.7, which can cause some campatibility issues between packages. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next [Chapter](./reproducibility.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "c446eef832ec964573dc49f36fd16bdbed40cbfbefbf557bc2dc78d9e7968689"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}