Merge pull request #66 from ShervanGharari/develop_2.0.0

Develop 2.0.0

.gitignore

@@ -7,3 +7,164 @@
 /examples/fig/
 /examples/__pycache__/
 /examples/.*/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

.readthedocs.yaml

@@ -0,0 +1,17 @@
+version: "2"
+
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.9"
+
+
+# Build from the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Explicitly set the version of Python and its requirements
+python:
+  install:
+    - requirements: docs/requirements.txt
+

MANIFEST.in

@@ -1,2 +1,3 @@
 # Include the license file
 include LICENSE
+include src/easymore/scripts/assets/default.slurm

README.md

@@ -9,6 +9,8 @@ EASYMORE is a collection of functions that allows extraction of the data from a
 
 EASYMORE is very efficient as it uses pandas `groupby` functionality. Remapping of the entire north American domain from ERA5 with resolution of 0.25 degree to 500,000 subbasins of MERIT-Hydro watershed for 7 variables in 1.2 seconds for one time step (the time varying from device to device and depending on the source netCDF files sizes and their temporal aggregation).
 
+EASYMORE also allows parallel computing across many netCDF files as well as command line interface to easily interact with its `nc_remapper()` functionallity.
+
 ## The code can be used for the following purposes:
 
 1. Remapping the relevant forcing variables, such as precipitation or temperature and other variables for the effortless model set up. This transfer can be from Thiessen polygon or gridded data, for example, to computational units, hydrological model for example.
@@ -37,7 +39,11 @@ pip install .
 
 ## Flexibilities:
 
-1. EASYMORE allow for commbination of the remapping of NetCDF on local computer or remote high performance computer. For example, the the GIS steps of creating remapping file can be done locally on a sample file that contains few time step of the data (but all the domain). EASYMORE can then be directed to remapping file on the HPC and will skip all the needed GIS steps and directly start remapping process of bulk of the data.
+1. EASYMORE allows for commbination of the remapping of NetCDF on local computer or remote high performance computer. For example, the the GIS steps of creating remapping file can be done locally on a sample file that contains few time step of the data (but all the domain). EASYMORE can then be directed to remapping file on the HPC and will skip all the needed GIS steps and directly start remapping process of bulk of the data.
+
+2. EASYMORE allows for parallel remapping of many NetCDF files on local computer or HPC.
+
+3. Commnad line interface to easily call EASYMORE nc_remapper functionality while populating the varibales directly or from a saved config file.
 
 ## Examples:
 
@@ -48,7 +54,9 @@ pip install .
 5. [Remap variables from an irregular shapefile data from Thiessen polygons of station data to irregular shapes.](https://github.com/ShervanGharari/EASYMORE/blob/main/examples/04_Thiessen_Polygons_SCDNA.ipynb)
 6. [Remap variables from irregular shapefile data, such as administrative boundaries for example, to irregular shapes.](https://github.com/ShervanGharari/EASYMORE/blob/main/examples/05_irregular_shapes.ipynb)
 7. [Extract variables for points (such as locations of stations, cities, etc) from the grided or irregular shapefiles; temperature example](https://github.com/ShervanGharari/EASYMORE/blob/main/examples/07_GRACE_capitals.ipynb)
-8. [Interpolate regular, rotated, or irregular data and model output to any resolution density of the points by reconstructing the remapping file](https://github.com/ShervanGharari/EASYMORE/blob/main/examples/92_WRF_interpolation.ipynb)
+8. [Parallel remapping of various NetCDF files from source to remapped on local computer](https://github.com/ShervanGharari/EASYMORE/blob/main/examples/11a_parallel_no_job.ipynb) or [HPC](https://github.com/ShervanGharari/EASYMORE/blob/main/examples/11b_parallel_job.py) with [SLURM scheduler](https://github.com/ShervanGharari/EASYMORE/blob/main/examples/11b_parallel_job.slurm)
+
+## Illustrative visualization:
 
 The two figures show remapping of the gridded temperature from ERA5 data set to subbasin of South Saskatchewan River at Medicine Hat.
 
@@ -60,12 +68,39 @@ The two figures show remapping of the gridded temperature from ERA5 data set to
 
 ![](https://github.com/ShervanGharari/EASYMORE/blob/main/fig/Remapped_new.png)
 
+## How to cite:
+
+```
+@article{gharari_easymore_2023,
+	title = {{EASYMORE}: {A} {Python} package to streamline the remapping of variables for {Earth} {System} models},
+	volume = {24},
+	issn = {2352-7110},
+	shorttitle = {{EASYMORE}},
+	url = {https://www.sciencedirect.com/science/article/pii/S2352711023002431},
+	doi = {10.1016/j.softx.2023.101547},
+	urldate = {2023-11-07},
+	journal = {SoftwareX},
+	author = {Gharari, Shervan and Keshavarz, Kasra and Knoben, Wouter J. M. and Tang, Gouqiang and Clark, Martyn P.},
+	month = dec,
+	year = {2023},
+	keywords = {EASYMORE, Earth System modeling, NetCDF, Remapping, Shapefile},
+	pages = {101547},
+}
+```
+
+Link to the above [publication](http://dx.doi.org/10.1016/j.softx.2023.101547).
+
 ## Publication that have used EASYMORE so far:
 
-Knoben, W. J. M., Clark, M. P., Bales, J., Bennett, A., Gharari, S., Marsh, C. B., Nijssen, B., Pietroniro, A., Spiteri, R. J., Tarboton, D. G., Wood, A. W.: Community Workflows to Advance Reproducibility in Hydrologic Modeling: Separating Model-Agnostic and Model-Specific Configuration Steps in Applications of Large-Domain Hydrologic Models, Water Resources Research, 58, e2021WR031753. https://doi.org/10.1029/2021WR031753
+Tang, G., Clark, M. P., Knoben, W. J. M., Liu, H., Gharari, S., Arnal, L., Beck, H. E., Wood, A. W., Newman, A. J., Papalexiou, S. M. The impact of meteorological forcing uncertainty on hydrological modeling: A global analysis of cryosphere basins. Water Resources Research, 59, e2022WR033767. https://doi.org/10.1029/2022WR033767, 2023.
 
-Gharari, S., Vanderkelen, I., Tefs, A., Mizukami, N., Stadnyk, T. A., Lawrence, D., Clark, M. P.: A Flexible Multi-Scale Framework to Simulate Lakes and Reservoirs in Earth System Models, Earth and Space Science Open Archive, 24, https://doi.org/10.1002/essoar.10510902.1, 2022
+Knoben, W. J. M., Clark, M. P., Bales, J., Bennett, A., Gharari, S., Marsh, C. B., Nijssen, B., Pietroniro, A., Spiteri, R. J., Tarboton, D. G., Wood, A. W.: Community Workflows to Advance Reproducibility in Hydrologic Modeling: Separating Model-Agnostic and Model-Specific Configuration Steps in Applications of Large-Domain Hydrologic Models, Water Resources Research, 58, e2021WR031753. https://doi.org/10.1029/2021WR031753, 2022.
 
-Gharari, S., Clark, M. P., Mizukami, N., Knoben, W. J. M., Wong, J. S., and Pietroniro, A.: Flexible vector-based spatial configurations in land models, Hydrol. Earth Syst. Sci., 24, 5953–5971, https://doi.org/10.5194/hess-24-5953-2020, 2020
+Gharari, S., Vanderkelen, I., Tefs, A., Mizukami, N., Stadnyk, T. A., Lawrence, D., Clark, M. P.: A Flexible Multi-Scale Framework to Simulate Lakes and Reservoirs in Earth System Models, Earth and Space Science Open Archive, 24, https://doi.org/10.1002/essoar.10510902.1, 2022.
+
+Li, Z., Gao, S., Chen, M., Gourley, J., Mizukami, N., and Hong, Y.: CREST-VEC: a framework towards more accurate and realistic flood simulation across scales, Geosci. Model Dev., 15, 6181–6196, https://doi.org/10.5194/gmd-15-6181-2022, 2022.
 
 Sheikholeslami, R., Gharari, S., Papalexiou, S. M., Clark, M. P.: VISCOUS: A Variance-Based Sensitivity Analysis Using Copulas for Efficient Identification of Dominant Hydrological Processes, Water Resources Research, https://doi.org/10.1029/2020WR028435, 2021.
+
+Gharari, S., Clark, M. P., Mizukami, N., Knoben, W. J. M., Wong, J. S., and Pietroniro, A.: Flexible vector-based spatial configurations in land models, Hydrol. Earth Syst. Sci., 24, 5953–5971, https://doi.org/10.5194/hess-24-5953-2020, 2020.
+

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/conf.py

@@ -0,0 +1,40 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+# paths from conf.py to top directory and where package is located
+import sys
+import os
+#sys.path.insert(0, os.path.abspath('../src/easymore/'))
+sys.path.insert(0, os.path.abspath('../src/'))
+#sys.path.insert(0, os.path.abspath('..'))
+
+project = 'EASYMORE'
+copyright = '2023, Shervan Gharari'
+author = 'Shervan Gharari'
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+# Extensions from conf.py
+extensions = [
+    'sphinx.ext.autodoc',	        # To generate autodocs
+    'sphinx.ext.mathjax',           # autodoc with maths
+    'sphinx.ext.napoleon'           # For auto-doc configuration
+]
+
+napoleon_google_docstring = False
+napoleon_numpy_docstring = True
+
+templates_path = ['_templates']
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = 'sphinx_rtd_theme'
+html_static_path = ['_static']

docs/index.rst

@@ -0,0 +1,21 @@
+.. EASYMORE documentation master file, created by
+   sphinx-quickstart on Sun Aug 27 16:45:29 2023.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to EASYMORE's documentation!
+====================================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   modules
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd