Merging with main. This is completely untested, I just fixed merge co…

…nflicts. I hope to test this soon and get the PR back up to date with redstreet's suggestions

.github/workflows/conventionalcommits.yml

@@ -0,0 +1,15 @@
+name: Conventional Commits
+
+on:
+  pull_request:
+    branches: [ main ]
+    types: [opened, reopened, edited, synchronize]
+
+jobs:
+  build:
+    name: Conventional Commits
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: webiny/[email protected]

.github/workflows/pythonpackage.yml

@@ -8,6 +8,7 @@ on:
     branches: [ main ]
   pull_request:
     branches: [ main ]
+    types: [opened, reopened, edited, synchronize]
 
 jobs:
   build:
@@ -16,7 +17,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.8", "3.9", "3.10"]
+        python-version: ["3.11"]
 
     steps:
     - uses: actions/checkout@v3
@@ -31,6 +32,10 @@ jobs:
     - name: Lint with ruff
       run: |
         ruff check . --statistics
-    # - name: Test with pytest
-    #   run: |
-    #     pytest
+    - name: Test with pytest
+      run: |
+        pytest
+    - name: Check formatting is applied
+      run: |
+        ruff format --check
+        isort --profile black --check .

.gitignore

@@ -50,6 +50,7 @@ coverage.xml
 *.py,cover
 .hypothesis/
 .pytest_cache/
+.debug-*
 
 # Translations
 *.mo

CHANGELOG.md

@@ -1,6 +1,48 @@
 # Changelog
 
-## (unreleased)
+## 0.9.0 (2024-05-14)
+
+### New Features
+
+- add genericpdf paycheck importer. [Ammon Sarver]
+- add bamboohr paycheck importer. [Ammon Sarver]
+- add pdfreader libreader importer. [Ammon Sarver]
+- minor: add 'show_configured' in paycheck transaction builder. [Red S]
+
+### New Importers
+- add CSV importer for Digital Credit Union (#89) [Harlan Lieberman-Berg]
+- add importer for Alliant Federal Credit Union (#88) [Harlan Lieberman-Berg]
+- add schwab csv credit line importer. [Red S]
+
+### Improvements
+
+- minor: add_custom_postings to banking.py. [Red S]
+- minor: add identification based on filename to schwab importers. [Red S]
+- minor: overridable add_custom_postings() [Red S]
+- minor: add tdameritrade to template.cfg. [Red S]
+
+
+### Fixes
+
+- update requirements to add back lost packages. [Ammon Sarver]
+- schwab_csv_creditline balance sign. [Red S]
+- add file encoding support to csvreader. [Ad Timmering]
+- only emit filing account metadata if configured #97. [Red S]
+- one-file-per-account broke with smart_importer #97. [Red S]
+- timestamp issue in balance assertions. [Red S]
+- balance date on test. [Red S]
+- schwab tests. [Red S]
+- schwab doesn't use a header in their csv any more. [Red S]
+- schwab_csv_checking format changed #91. [Red S]
+
+### Other
+- enforce formatting with ruff. [Rane Brown]
+- format with isort, use pyproject.toml. [Rane Brown]
+- switch to ruff for linting (#90) [Harlan Lieberman-Berg]
+- style: reformat to 99 col width (previously 88 col) [Red S]
+
+
+## 0.8.0 (2023-11-24)
 
 
 ### Improvements

CONTRIBUTING.md

@@ -1,3 +1,5 @@
+# Contributing
+
 Contributions welcome. Preferably:
 - include a test file. I realize this is sometimes a pain to create, but there is no way
   for me to test external contributions without test files
@@ -18,3 +20,30 @@ Contributions welcome. Preferably:
                ├── History_for_Account_X8YYYYYYY.csv
                └── run_test.bash
   ```
+
+## Setup
+
+Development setup would typically look something like this:
+
+```bash
+# clone repo, cd to repo
+
+# create virtual environment
+python3 -m venv venv
+
+# activate virtual environment
+source venv/bin/activate
+
+# install dependencies
+pip install -e .[dev]
+```
+
+## Formatting
+
+Prior to finalizing a pull request make sure to run the formatting tools and
+commit any resulting changes.
+
+```bash
+ruff format
+isort --profile black .
+```

README.md

@@ -32,6 +32,8 @@ File format readers included are:
 - `.ofx`
 - `.csv` (single and multitable support)
 - `.xlsx` (single and multitable support) (`pip3 install xlrd` if you plan to use this)
+- `.pdf` (single and multitable support)
+
 
 Transaction builders included are:
 - Banking (for banks and credit cards, which benefit from a postings predictor like
@@ -98,7 +100,8 @@ for f in ~/.zcomplete/*; do source $f; done
 pip3 install beancount-reds-importers
 ```
 
-Or to install the bleeding edge version from git:
+Or to install the bleeding edge version from git (which I recommend, as long as you are
+willing to understand there might be a bug or two):
 ```
 pip3 install git+https://github.com/redstreet/beancount_reds_importers
 ```
@@ -133,8 +136,12 @@ If you want something else, simply override this method in individual importer
 
 `smart` dates: Banks and credit cards typically have pending transactions that are not
 included in downloads. When we download the next statement, new transactions may appear
-prior to the balance assertion date that we generate for this statement. To attempt to
-avoid this, we set the balance assertion date to either two days (fudge factor to
+prior to the balance assertion date that we generate for this statement, which renders
+this balance assertion invalid. This problem manifests occasionally as an existing
+balance statement breaking  when a new statement is downloaded and is an annoyance as it
+needs manual fixing.
+
+To minimize this, we set the balance assertion date to either two days (fudge factor to
 account for pending transactions) before the statement's end date or the last
 transaction's date, whichever is later. To choose a different fudge factor, simply set
 `balance_assertion_date_fudge` in your config.

beancount_reds_importers/example/fund_info.py

@@ -20,15 +20,15 @@
 # mutual funds since those are brokerage specific.
 
 fund_data = [
- ('SCHF',   '808524805', 'Schwab International Equity ETF'),
- ('VGTEST', '012345678', 'Vanguard Test Fund'),
- ('VMFXX',  '922906300', 'Vanguard Federal Money Market Fund'),
+    ("SCHF", "808524805", "Schwab International Equity ETF"),
+    ("VGTEST", "012345678", "Vanguard Test Fund"),
+    ("VMFXX", "922906300", "Vanguard Federal Money Market Fund"),
 ]
 
 # list of money_market accounts. These will not be held at cost, and instead will use price conversions
-money_market = ['VMFXX']
+money_market = ["VMFXX"]
 
 fund_info = {
-        'fund_data': fund_data,
-        'money_market': money_market,
-        }
+    "fund_data": fund_data,
+    "money_market": money_market,
+}

beancount_reds_importers/example/my-smart.import

@@ -4,7 +4,8 @@
 import sys
 from os import path
 
-from smart_importer import apply_hooks, PredictPayees, PredictPostings
+from smart_importer import PredictPayees, PredictPostings, apply_hooks
+
 sys.path.insert(0, path.join(path.dirname(__file__)))
 
 from beancount_reds_importers.importers import ally

beancount_reds_importers/example/my.import

@@ -6,9 +6,11 @@ from os import path
 
 sys.path.insert(0, path.join(path.dirname(__file__)))
 
+from fund_info import *
+
 from beancount_reds_importers.importers import vanguard
 from beancount_reds_importers.importers.schwab import schwab_csv_brokerage
-from fund_info import *
+
 # For a better solution for fund_info, see: https://reds-rants.netlify.app/personal-finance/tickers-and-identifiers/
 
 # Setting this variable provides a list of importer instances.

beancount_reds_importers/importers/alliant/__init__.py

@@ -5,10 +5,10 @@
 
 
 class Importer(banking.Importer, ofxreader.Importer):
-    IMPORTER_NAME = 'Alliant Credit Union'
+    IMPORTER_NAME = "Alliant Credit Union"
 
     def custom_init(self):
         if not self.custom_init_run:
             self.max_rounding_error = 0.04
-            self.filename_pattern_def = '.*alliant'
+            self.filename_pattern_def = ".*alliant"
             self.custom_init_run = True

beancount_reds_importers/importers/ally/__init__.py

@@ -5,10 +5,10 @@
 
 
 class Importer(banking.Importer, ofxreader.Importer):
-    IMPORTER_NAME = 'Ally Bank'
+    IMPORTER_NAME = "Ally Bank"
 
     def custom_init(self):
         if not self.custom_init_run:
             self.max_rounding_error = 0.04
-            self.filename_pattern_def = '.*transactions'
+            self.filename_pattern_def = ".*transactions"
             self.custom_init_run = True

beancount_reds_importers/importers/ally/tests/ally_test.py

@@ -1,5 +1,7 @@
 from os import path
+
 from beancount.ingest import regression_pytest as regtest
+
 from beancount_reds_importers.importers import ally
 
 
@@ -8,6 +10,7 @@
         {
             "account_number": "23456",
             "main_account": "Assets:Banks:Checking",
+            "emit_filing_account_metadata": False,
         }
     )
 )

beancount_reds_importers/importers/amazongc/__init__.py

@@ -23,9 +23,10 @@
 import datetime
 import itertools
 import ntpath
+
 from beancount.core import data
-from beancount.ingest import importer
 from beancount.core.number import D
+from beancount.ingest import importer
 
 # account flow                          ingest source
 # ----------------------------------------------------
@@ -37,25 +38,25 @@
 class Importer(importer.ImporterProtocol):
     def __init__(self, config):
         self.config = config
-        self.currency = self.config.get('currency', 'CURRENCY_NOT_CONFIGURED')
-        self.filename_pattern_def = 'amazon-gift-card.tsv'
+        self.currency = self.config.get("currency", "CURRENCY_NOT_CONFIGURED")
+        self.filename_pattern_def = "amazon-gift-card.tsv"
 
     def identify(self, file):
         return self.filename_pattern_def in file.name
 
     def file_name(self, file):
-        return '{}'.format(ntpath.basename(file.name))
+        return "{}".format(ntpath.basename(file.name))
 
     def file_account(self, _):
-        return self.config['main_account']
+        return self.config["main_account"]
 
     def file_date(self, file):
         "Get the maximum date from the file."
         maxdate = datetime.date.min
-        for line in open(file.name, 'r').readlines()[1:]:
-            f = line.split('\t')
+        for line in open(file.name, "r").readlines()[1:]:
+            f = line.split("\t")
             f = [i.strip() for i in f]
-            date = datetime.datetime.strptime(f[0], '%B %d, %Y').date()
+            date = datetime.datetime.strptime(f[0], "%B %d, %Y").date()
             maxdate = max(date, maxdate)
         return maxdate
 
@@ -65,18 +66,26 @@ def extract(self, file, existing_entries=None):
         new_entries = []
 
         counter = itertools.count()
-        for line in open(file.name, 'r').readlines()[1:]:
-            f = line.split('\t')
+        for line in open(file.name, "r").readlines()[1:]:
+            f = line.split("\t")
             f = [i.strip() for i in f]
-            date = datetime.datetime.strptime(f[0], '%B %d, %Y').date()
+            date = datetime.datetime.strptime(f[0], "%B %d, %Y").date()
             description = f[1].encode("ascii", "ignore").decode()
-            number = D(f[2].replace('$', ''))
+            number = D(f[2].replace("$", ""))
 
             metadata = data.new_metadata(file.name, next(counter))
-            entry = data.Transaction(metadata, date, self.FLAG,
-                                     None, description, data.EMPTY_SET, data.EMPTY_SET, [])
-            data.create_simple_posting(entry, config['main_account'], number, self.currency)
-            data.create_simple_posting(entry, config['target_account'], None, None)
+            entry = data.Transaction(
+                metadata,
+                date,
+                self.FLAG,
+                None,
+                description,
+                data.EMPTY_SET,
+                data.EMPTY_SET,
+                [],
+            )
+            data.create_simple_posting(entry, config["main_account"], number, self.currency)
+            data.create_simple_posting(entry, config["target_account"], None, None)
             new_entries.append(entry)
 
         return new_entries

beancount_reds_importers/importers/amex/__init__.py

@@ -5,10 +5,10 @@
 
 
 class Importer(banking.Importer, ofxreader.Importer):
-    IMPORTER_NAME = 'American Express'
+    IMPORTER_NAME = "American Express"
 
     def custom_init(self):
         if not self.custom_init_run:
             self.max_rounding_error = 0.04
-            self.filename_pattern_def = '.*amex'
+            self.filename_pattern_def = ".*amex"
             self.custom_init_run = True