Squashed commit of the following:

commit 0f7a7e8 Author: Doug Ransom <[email protected]> Date: Fri Feb 2 09:33:50 2024 -0800 Fixed logging intialization in sample_plots deleted extraneous and incorrect powershell script test.ps1 Changed from 'flit build' to 'build' to build the package. commit 52f0c2d Author: Doug Ransom <[email protected]> Date: Sat Jul 22 12:10:45 2023 -0700 add vix1d commit 4e32d47 Author: Doug Ransom <[email protected]> Date: Wed Jul 19 14:03:43 2023 -0700 fixed gaps in 2013 futures. commit c971912 Author: Doug Ransom <[email protected]> Date: Tue Jul 18 09:24:22 2023 -0700 added a test to check for same dates on SHORTVOL and futures trade dates. commit 54bac5d Author: Doug Ransom <[email protected]> Date: Sat Jul 15 13:47:48 2023 -0700 checkpoint commit cd86c99 Author: Doug Ransom <[email protected]> Date: Sat Jul 15 10:49:49 2023 -0700 checkpoint commit 81737ee Author: Doug Ransom <[email protected]> Date: Sat Jul 15 10:20:45 2023 -0700 checkpoint commit c662987 Author: Doug Ransom <[email protected]> Date: Sat Jul 15 10:03:23 2023 -0700 checkpiont commit c1ea665 Author: Doug Ransom <[email protected]> Date: Sat Jul 15 10:02:11 2023 -0700 checkpoint commit 00435cc Author: Doug Ransom <[email protected]> Date: Sat Jul 15 09:24:16 2023 -0700 checkpoint commit b956c7f Author: Doug Ransom <[email protected]> Date: Sat Jul 15 06:23:23 2023 -0700 basis prototype commit 7210b0c Author: Doug Ransom <[email protected]> Date: Wed Jul 5 16:38:15 2023 -0700 updating commit 8cf936c Author: Doug Ransom <[email protected]> Date: Wed Jul 5 14:02:47 2023 -0700 updating commit 9d3577f Author: Doug Ransom <[email protected]> Date: Wed Jul 5 13:59:35 2023 -0700 updating from main commit c62789c Author: Doug Ransom <[email protected]> Date: Wed Jul 5 13:57:35 2023 -0700 updating from main commit e4258d6 Author: Doug Ransom <[email protected]> Date: Wed Jul 5 13:09:27 2023 -0700 python_publish.yml commit 198d93b Author: Doug Ransom <[email protected]> Date: Wed Jul 5 10:54:07 2023 -0700 Renamed function to be more appropriate. commit fe724b2 Author: Doug Ransom <[email protected]> Date: Wed Jul 5 10:51:21 2023 -0700 The monthly interpolation of expiry date is close to working. Same with the weighted front two months. A couple unit tests might reveal some subtly bugs. commit 4814544 Author: Doug Ransom <[email protected]> Date: Wed Jul 5 07:02:34 2023 -0700 checkpoint commit f6988a6 Author: Doug Ransom <[email protected]> Date: Wed Jul 5 06:04:14 2023 -0700 checkpoint. commit 7a8746c Author: Doug Ransom <[email protected]> Date: Tue Jul 4 16:41:03 2023 -0700 removed unused test file commit 60d0183 Author: Doug Ransom <[email protected]> Date: Tue Jul 4 16:38:02 2023 -0700 corrected weight calculations commit 43ca576 Author: Doug Ransom <[email protected]> Date: Tue Jul 4 14:24:27 2023 -0700 added ovx commit 1d0a8f5 Author: Doug Ransom <[email protected]> Date: Tue Jul 4 14:12:04 2023 -0700 added shortvol, longvol commit fe4eafb Author: Doug Ransom <[email protected]> Date: Tue Jul 4 14:06:18 2023 -0700 added vvix and VXTLT, GVZ to sample plot. commit 073fd88 Author: Doug Ransom <[email protected]> Date: Tue Jul 4 13:38:07 2023 -0700 . commit 78531b6 Author: Doug Ransom <[email protected]> Date: Tue Jul 4 13:21:58 2023 -0700 checkpoint, tests work when pandas_market_calendars work. commit 70233be Author: Doug Ransom <[email protected]> Date: Tue Jul 4 13:18:07 2023 -0700 checkpoint. commit 7f080d5 Author: Doug Ransom <[email protected]> Date: Tue Jul 4 12:24:02 2023 -0700 checkpoint
dougransom · Feb 2, 2024 · 6a4cca9 · 6a4cca9
1 parent cc23081
commit 6a4cca9
Show file tree

Hide file tree

Showing 18 changed files with 4,191 additions and 245 deletions.
diff --git a/.github/workflows/python_publish.yml b/.github/workflows/python_publish.yml
@@ -1,29 +1,25 @@
-name: Publish Package 📦 to Test PyPI
+name: Publish Package 📦 to  PyPI
 on:
   release:
     types: [published] # with prerelease and release
 
 permissions:
   contents: read
-
+  id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
 jobs:
-  publish:
-    # Set up the environment `CI` references the secret `TEST_PYPI_API_TOKEN` in repository settings
+  build_and_publish:
+    # Set up the environment `CI` references the secret `PYPI_API_TOKEN` in repository settings
     # https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#referencing-an-environment
     environment: CI
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
     - name: Installing build Dependencies
       run: |
-          python -m pip install --upgrade pip
-          python -m pip install flit
-    - name: Building package with flit
-      run: |
-            flit build
-    - name: Publishing 📦 to  PyPI
-     # Regarding building artifacts within Platform specific environment see https://github.com/pypa/gh-action-pypi-publish#non-goals
+        python -m pip install --upgrade pip
+        pip install build
+    - name: Build package
+      run: python -m build
+    - name: Publish package
       uses: pypa/gh-action-pypi-publish@release/v1
-      with:
-        password: ${{ secrets.PYPI_API_TOKEN }}
-        repository-url: https://upload.pypi.org/legacy/
+
diff --git a/README.md b/README.md
@@ -13,6 +13,7 @@ Vix Cash Data are downloaded from [CBOE Historical Volatility Indexes](https://w
 There is an API for Python to load the data into Pandas DataFrames.  If you do your analysis in Python, use the API.
 
 Since there is no documentation yet, look at the examples in the src/vix_utils/examples folder.
+There is a Jupyter Notebook vix_utils.ipynb in that folder.
 
 *Important note for Juypter notebooks.*  
 You must use  async_get_vix_index_histories and async_load_vix_term_structure 
@@ -69,6 +70,15 @@ However, new features and bug fixes should be developed with [Test Driven Develo
 ## Examples
 Source is in `src/vix_utils/examples`
 
+~~~
+## Data Notes
+These dates appear to be missing from the CBOE Data.
+At some point they need to be patched in if they exist.
+```
+[Timestamp('2006-11-10 00:00:00'), Timestamp('2007-01-03 00:00:00'), Timestamp('2021-04-02 00:00:00'), Timestamp('2021-12-24 00:00:00')]
+```
+There seem to be  a few dates where spot indexes are missing, you will have to workaround by using fill feature of Pandas datafame, or skip those days, in any analysis.
 ~~~
 ## Developing
 https://numpydoc.readthedocs.io/en/latest/format.html
+
diff --git a/pyproject.toml b/pyproject.toml
@@ -13,7 +13,7 @@ classifiers = ["License :: OSI Approved :: MIT License",
 	"Topic :: Office/Business :: Financial :: Investment"]
 
 readme="README.md"
-requires-python=">=3.11"
+requires-python=">=3.10"
 
 dynamic=["version"]
 
@@ -23,7 +23,8 @@ dependencies= [
     "aiofiles",
     "aiohttp[speedups]",
     "openpyxl",
-    "appdirs"]
+    "appdirs",
+    "more_itertools"]
 
 
 [project.optional-dependencies]

diff --git a/src/vix_utils/__init__.py b/src/vix_utils/__init__.py
@@ -2,15 +2,15 @@
 A library for preparing   VIX Futures and Cash Term Structures for analysis,
 including a  continuous maturity VIX Futures term structure.
 """
-__version__ = '0.1.4'
+__version__ = '0.1.5'
 
 from .download_vix_futures import  \
 pivot_futures_on_monthly_tenor,select_monthly_futures,async_load_vix_term_structure,load_vix_term_structure
 
 from .vix_cash_term_structure import \
     async_get_vix_index_histories,  \
     get_vix_index_histories,    \
-    pivot_cash_term_structure_on_symbol
+    pivot_spot_term_structure_on_symbol
 
 from .vix_futures_dates import vix_futures_expiry_date_monthly, \
     vix_futures_expiry_date_from_trade_date, \

diff --git a/src/vix_utils/continuous_maturity.py b/src/vix_utils/continuous_maturity.py
@@ -61,7 +61,7 @@ def do_weighting_front_two_months(trades_df : pd.DataFrame,weight_df : pd.DataFr
 def append_continuous_maturity_one_month(monthly_wide_records : pd.DataFrame)->pd.DataFrame:
     """
     produces a weighted mean of the two nearest monthly futures (using continous_maturity_30day)
-    appends it to the monthly_wide_records.
+    appends it to the monthly_wide_records, with Monthly_Tenor of 1.5 (for ease of sorting)
     There will be fewer columns as the result of continous_maturity_30day has fewer columns for a tenor than 
     monthly_wide_records.
     parameters:
@@ -81,11 +81,11 @@ def append_continuous_maturity_one_month(monthly_wide_records : pd.DataFrame)->p
     new_df2=new_df1.swaplevel(axis=1)
     #add a level to new_cm
     d={}
-    d["30 Day Continuous"]=new_cm
+    d[1.5]=new_cm
     e=pd.concat(d,axis=1)
 
     #concatenate new_cm (after adding the level of idexing)
-    new_df3=pd.concat([new_df2,e],axis=1)
+    new_df3=pd.concat([new_df2,e],axis=1).sort_index(axis=1)
     return new_df3
 
 

diff --git a/src/vix_utils/download_vix_futures.py b/src/vix_utils/download_vix_futures.py
@@ -116,10 +116,9 @@ def years_and_months():
 
 def archived_years_and_months():
     "For data from https://www.cboe.com/us/futures/market_statistics/historical_data/archive/"
-    #specifically avoid 2013 since the data is dirty and duplicated with the
-    #weekly and monthly data from the current download source.
+
 
-    return list(itertools.product(range(2004,2013),range(1,13)))
+    return list(itertools.product(range(2004,2014),range(1,13)))
 
 def years_and_weeks():
     now = dt.datetime.now()
@@ -320,7 +319,8 @@ def downloaded_file_paths(data_dir:Path)->tuple[Path,Path,Path]:
         folders_contents=tuple( list(the_dir.glob("*.csv")) for the_dir in (a,b,c))
 
         return folders_contents  
-
+_header_match_str="Trade Date,"
+_head_match_len=len(_header_match_str)
 async def download(vixutil_path:Path):
     """
     Download the vix futures historis we don't have up todate.
@@ -338,14 +338,20 @@ async def download(vixutil_path:Path):
 
 
         #need to find lines with a trailing "," and remove it.  There are a bunch in the 
-        #archived data
+        #archived data.
+        #we also need to throw away lines before the line that starts with Trade Date.
         _,_,amfns=downloaded_file_paths(vixutil_path)
+
+        def unmatched_header_row(a_line):
+            return a_line[0:_head_match_len]!=_header_match_str
+
         for fn in amfns:
             with open(fn,'r') as fin:
                 data=fin.read().splitlines(True)
 
+            filtered_preamble=itertools.dropwhile(unmatched_header_row,data)
             with open(fn,'w') as fout:
-                for line in data:
+                for line in filtered_preamble:
                     updated_line=",".join(line.split(",")[0:11]).strip()
                     print(updated_line,file=fout)
 
@@ -400,8 +406,9 @@ def read_csv_future_file(future_path:Path,monthly_expiry_date_strings:frozenset)
     try:
         df = pd.read_csv(future_path,parse_dates=[0])
     except Exception as e:
-        logging.warn(f"\n {e}\n reading\n{future_path} ")
-        raise
+        logging.warn(f"\n{e}\n reading\n{future_path}, skipping ")
+        return pd.DataFrame(columns=[["Trade Date"]])
+
     fn=future_path.name
     expiry_date_str=settlement_date_str_from_fn(fn)
     #week_number  TODO FIGURE THIS OUT
@@ -551,8 +558,9 @@ def load_vix_term_structure(forceReload=False):
     last call, if one exists.
 
     """
-    with asyncio.Runner() as runner:
-        return runner.run(async_load_vix_term_structure(forceReload))
+    return asyncio.run(async_load_vix_term_structure(forceReload))
+#    with asyncio.Runner() as runner:
+#        return runner.run(async_load_vix_term_structure(forceReload))
 
 @timeit()    
 async def async_load_vix_term_structure(forceReload=False)->pd.DataFrame:

diff --git a/src/vix_utils/examples/a_t.ipynb b/src/vix_utils/examples/a_t.ipynb
@@ -0,0 +1,43 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "1631065c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import xarray as xa, numpy as np"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f612f82b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/src/vix_utils/examples/hello_x.py b/src/vix_utils/examples/hello_x.py
@@ -0,0 +1,7 @@
+import xarray as xr, pandas as pd, numpy as np
+
+data = xr.DataArray(np.random.randn(2, 3), dims=("x", "y"), coords={"x": [10, 20]})
+print(f"data{data}")
+a = xr.DataArray(np.random.randn(3), [data.coords["y"]])
+b = xr.DataArray(np.random.randn(4), dims="z")
+3
diff --git a/src/vix_utils/examples/sample_load_data.py b/src/vix_utils/examples/sample_load_data.py
@@ -5,10 +5,11 @@
 
 import logging
 import sys
+from itertools import chain
 stars='*'*80
-def pstars():
+def pstars(toprint=""):
     """Print a line of '*' """ 
-    print(stars)
+    print(f"\n{stars}\n{toprint}")
 
 def main():
     logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
@@ -33,7 +34,7 @@ async def do_load():
     vix_futures_history=v.load_vix_term_structure()
 
     pstars()
-    wide_cash=v.pivot_cash_term_structure_on_symbol(vix_cash_history)
+    wide_cash=v.pivot_spot_term_structure_on_symbol(vix_cash_history)
     print(f"Wide Vix Cash history\n{wide_cash}\nWide Cash History Index\n{wide_cash.columns}")
 
     pstars()
@@ -51,23 +52,68 @@ async def do_load():
     pstars()
     pivoted= v.pivot_futures_on_monthly_tenor(monthly)
     print(f"\npivoted {pivoted}")
+
+    pstars() 
+    indexed_by_tenor=vix_futures_history.set_index(["Trade Date","Tenor_Monthly"])
+    print(f"indexed by tenor:\n{indexed_by_tenor}")
+
+
     pivoted_swapped=pivoted.swaplevel(0,1,axis=1)
 
     pivoted_two_cols=pivoted_swapped[['Close','File']]
-    print(f"The monthlys, with a tenor column index, levels swapped, just a few columns:\n{pivoted_two_cols}\ncolumn_index{pivoted_two_cols.columns}")
+    olhc=["Open","High","Low","Close"]
+    pivoted_ohlc=pivoted_swapped[olhc]
+    vix_ohlc=wide_cash.swaplevel(0,1,axis=1)[["VIX"]].swaplevel(0,1,axis=1)
+    #get the columns correspondenting to futures tenors
 
-    pstars()
+    #replicate in the spot prices
+
+
+
+
     m1m2_weighted=v.continuous_maturity_one_month(pivoted)
-    print(f"\nm1m2 weighted:\n{m1m2_weighted}\ncolumns:\n{m1m2_weighted.columns}")
+    pstars(f"\nm1m2 weighted:\n{m1m2_weighted}\ncolumns:\n{m1m2_weighted.columns}")
 
     appended_m1m2=v.append_continuous_maturity_one_month(pivoted)
-    appended_m1m2_close=appended_m1m2[[1,'30 Day Continuous',2]].swaplevel(axis=1)[['Close','Tenor_Days','Expiry']]
-    pstars()
+    appended_m1m2_close=appended_m1m2[[1,1.5,2]].swaplevel(axis=1)[['Close','Tenor_Days','Expiry']]
+    pstars(f"\nappended_m1m2:\n{appended_m1m2}")
+
+    closes=appended_m1m2.swaplevel(axis=1)["Close"]
+
+    pstars(f"\ncloses\n{closes}")
     print(f"\nappended m1m2 to wide (close):\n{appended_m1m2_close}")
-    pstars()
+
+
+    vix_cash_history_closes=wide_cash["Close"]
+    spot_symbols=["VIX9D","VIX","VIX3M","GVZ"]  #some symbols to compare with the VIX futures for a basis.
+                                                #the basis that makes sense of course is on the VIX, since the vix futures 
+                                                #are for the VIX spot settlement.
+                                                #there reasonably should be a relationship with the various
+                                                #vix spot indexes and probably a weaker one with GVZ.
+
+
+
+    def add_column_level(df:pd.DataFrame,var_name):
+        df2=pd.DataFrame(df)
+        idx=df2.columns.to_frame()
+        idx.insert(0,"Variable",var_name)
+        df2.columns=pd.MultiIndex.from_frame(idx)
+        return df2 
+
+    vix_basis_by_index=[add_column_level(closes.sub(vix_cash_history_closes[spot_symbol],axis=0),spot_symbol+"_Basis") for spot_symbol in ["VIX9D","VIX","VIX3M","GVZ"]]
+
+    closes=add_column_level(closes,"Futures")
+
+    vix_basis=pd.concat(chain([closes],vix_basis_by_index),axis=1,join="inner")
+
+    pstars(f"vix_basis{vix_basis}")
+
+
+
     with pd.option_context("display.max_rows",None,"display.max_columns",None):   
-        df2021_02=appended_m1m2.loc['2021-02'][[1,'30 Day Continuous',2]].swaplevel(axis=1)[['Close','Tenor_Days','Expiry']]
+        df2021_02=appended_m1m2.loc['2021-02'][[1,1.5,2]].swaplevel(axis=1)[['Close','Tenor_Days','Expiry']]
         print(f"\nappended (2021-02)\n{df2021_02}")
 
+
 if __name__=="__main__":
     main()