Skip to content

Commit

Permalink
Add time series models library
Browse files Browse the repository at this point in the history
  • Loading branch information
David Stuebe committed May 31, 2023
1 parent 280e2d1 commit 54699a4
Show file tree
Hide file tree
Showing 83 changed files with 19,033 additions and 1 deletion.
160 changes: 160 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright [2023] [Camus Energy]
Copyright 2023 Camus Energy

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
77 changes: 77 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Time Series Models

A library for composable forecasting models built on top of scikit-learn

The API for fit, predict, and metrics is reduced to specifying a start and end times for a given location.
The model must construct feature data using column transforms. Having done so, forecasting as a service become trivial.

## Installation

requires the eccodes library: `apt-get install libeccodes-dev`

`pip3 install -r requirements.txt`

Verify you can run the unit tests `python -m unittest`

If running in a container, to start a notebook use `jupyter notebook --NotebookApp.ip=0.0.0.0`

## Usage
Models can be composed of mixins for various estimators and forecast processes.

An ordinary least squares model for a Balancing Area Hourly Load
```python
class NpOlsModel(BalancingAreaHourly, LinearRegressor, RegularTimeSeriesModel):
pass
```
An XGBoost model for Balancing Area Hourly Load
```python
class XgbModel(BalancingAreaHourly, XgbRegressor, RegularTimeSeriesModel):
pass
```

The initialization arguments control feature construction.


### Trained models should be fully serializable using Cloud Pickle for maximum flexibility


It is easy to work with time series models in a
[Colab Notebook](https://colab.research.google.com/drive/1Tpoxdyf7aN1kyPrPb0L4uaZrmHQnHyB1?usp=sharing)


## Contents

* `data_fetchers/`: data fetchers and associated tests -- see directory-specific README.
* `back_test.py`: BackTest class for estimating model performance on out-of-sample data using a time series (rolling)
cross-validation.
* `config.py`: ConfigHandler helper class to encode and decode time series models configurations
* `constants.py`: defines constant variables referenced throughout Time Series Models library
* `data_monitor.py`: ForecastDataMonitor is a pipeline element that collects statistics on the data passing through it
* `decorators.py`: functions that can be used as decorators in Time Series Models library
* `dummy_models.py`: classes implementing `DummyDataModel` for using external predictions with Time Series Models
metrics and vizualizations
* `estimators.py`: Estimator and EstimatorWrapper mixins for forecast models (e.g., `XgbRegressor`, `RandomizedSearch`)
* `filters.py`: Filter mixins (e.g., `DropFeatures`, `RowFilter`, `StandardScaler`)
* Gap solving tools:
* `gap_maker.py`: utility for introducing synthetic gaps into a dataset
* `gap_filler.py`: utility for filling data gaps with various interpolative or extrapolative methods
* `gap_runner.py`: holds score_df_at_locs, a helper to evaluate gap-filling performance
* `metrics_runner.py`: a container for running models and comparing metrics, organized around monthly analysis
* `processes.py`: mixins defining the data sources and preprocessing steps for a given process to forecast
* `pv_physical_model.py`: common interface to using pvlib and PySAM physical models
* visualization tools:
* `shap_viz.py`: wrapper for applying `shap` library to RegularTimeSeriesModel instances
* `viz.py`: methods for plotting time series of predicted vs actual, and residual scatter plots
* `time_series_models.py`: core `RegularTimeSeriesModel` and `Mixin` class definitions, along with several other model
helpers and mixins like `AggModel` and `MonthSelector`.
* `time_unit.py`: Enum for calendar conversion of numpy datetime64 arrays
* transformers: a collection of transformers for composing model features
* `transformers.py`: broad-purpose (or uncategorized) transformers for use throughout Time Series Models
* core helper functions like `make_domain`
* some bespoke transformers that haven't been split out into separate modules
* simple array transformations
* `transformers_calendar.py`: calendar feature transformers (e.g., business day, day-of-week, & harmonics)
* `transformers_pv.py`: transformers for PV forecasting
* `transformers_shortcuts.py`: help avoid copy-pasting the same core features for each process
(forecast sum of meters & DER, rather than sum of forecasts)
* `version.py`: Time Series Models library version
Loading

0 comments on commit 54699a4

Please sign in to comment.