Skip to content

Latest commit

 

History

History
549 lines (395 loc) · 31.2 KB

CONTRIBUTING.md

File metadata and controls

549 lines (395 loc) · 31.2 KB
Raw

Contributing to Portfolio Performance

Contents

Development setup

Install Eclipse

  • Java 17, for example from Azul

  • Eclipse IDE - PP is build using the Eclipse RCP (Rich Client Platform) framework. Therefore it generally does not make sense to use other IDEs. Download the Eclipse IDE for RCP and RAP Developers package.

Optionally, install language packs for Eclipse:

  • Menu --> Help --> Install New Software
  • Use the following update site: 
    https://download.eclipse.org/technology/babel/update-site/latest/
  • Select the language packs you want to install
  • By default, Eclipse uses the host operating system language (locale). To force the use of another language, use the -nl parameter: 
    eclipse.exe -nl de

Install Eclipse plug-ins

Optionally, install via the Eclipse Marketplace (drag and drop the Install button to your Eclipse workspace)

Install from the simulatenous release update site

Menu --> Help --> Install New Software

Pick Latest Eclipse Simultaneous Release from the dropdown menu.

  • M2E PDE Integration (skip if you installed the Eclipse IDE for RCP and RAP Developers)
    • Under General Purpose Tools select the M2E PDE Integration
  • Eclipse e4 Tools Developer Resources
    • Under General Purpose Tools select the Eclipse e4 Tools Developer Resources

Configure Eclipse

Configure the following preferences (Menu --> Window --> Preferences)

  • Java --> Editor --> Save Actions
    • Activate Format Source Code and then Format edited lines
    • Activate Organize imports
  • Java --> Editor --> Content Assist
    • Activate Add import instead of qualified name
    • Activate Use static imports
  • Java --> Editor --> Content Assist --> Favorites
    • Click on New Type... and add the following favorites
      • name.abuchen.portfolio.util.TextUtil
      • name.abuchen.portfolio.datatransfer.ExtractorUtils
      • name.abuchen.portfolio.datatransfer.ExtractorMatchers
      • name.abuchen.portfolio.datatransfer.ExtractorTestUtilities
      • name.abuchen.portfolio.junit.TestUtilities
  • Java --> Installed JREs
    • Add the Java 17 JDK

Project setup

For further disucssion, check out the thread in the (German) Forum.

Source code

To contribute to Portfolio Performacne, you create a fork, clone the repository, make and push changes to your repository, and then create a pull request.

Setup Target Platform

  • Open the portfolio-target-definition project
  • Open the portfolio-target-definition.target file with the Target Editor (this may take a while as it requires Internet access). If you just get an XML file, use right click and chose Open With Target Editor
  • In the resulting editor, click on the "Set as Active Target Platform" link at the top right (this may also take a while)

Launch Portfolio Performance

PP uses Eclipse Launch Configuration DSL to define Eclipse launch configurations in a OS independent way.

First, add the Launch Configuration view to your workspace:

  • Menu --> Window --> Show View --> Other... --> Debug --> Launch Configuration

To run the application, select Eclipse Application --> PortfolioPerformance and right-click Run.

To run the tests, select under JUnit Plug-in Tests --> PortfolioPerformance_Tests or PortfolioPerformance_UI_Tests.

Build with Maven

It is not required to use Maven as you can develop using the Eclipse IDE with the setup above. The Maven build is used for the Github Actions build.

The Maven build works fine when JAVA_HOME points to an (Open-)JDK 17 installation.

Linux/macOS

export MAVEN_OPTS="-Xmx4g"
mvn -f portfolio-app/pom.xml clean verify

set MAVEN_OPTS="-Xmx4g"
mvn -f portfolio-app\pom.xml -Denforcer.skip=true clean verify

Hint: if you run into resolution problems, try deleting the ~/.m2/repository/p2 directory. Apparently, when switching to Maven Tycho 3, there are some layout changes.

Contribute code

  • Write a good commit message in English
  • If the change is related to a Github issue, add a line Closes #<ISSUE NUMBER> after an empty line
  • If the change is related to an thread in the forum, add a line Issue: https://... with the link to the post in the forum
  • Format the source code. The formatter configuration is part of the project source code. Exception: Do not reformat the PDF importer source code. Instead, carefully insert new code into the existing formatting.
  • Add test cases where applicable. Today, there are no tests that test the SWT UI. But add tests for all calculations.
  • Do not merge the the master branch into your feature branch. Instead, rebase your local changes to the head of the master branch.
  • Create a Pull Request - for example using GitHub Desktop using this tutorial

Translations

Currently, Portfolio Performance is translated into 13 languages.

To contribute a new language or assist with translations:

  • Register with the POEditor project.
  • Open a ticket if you plan to a add a new language and need test builds.
  • Before we create a new release, we download and merge the updated translations into the code base.

If you are contributing code:

Label descriptions

For naming externalized labels:

  • Use Label, Msg, Column as common prefixes for short labels, longer messages, and column headers respectively.
  • Use specific prefixes like PDF, CSV, Preferences, etc. if the area is big and distinct enough, but avoid creating new areas.
  • Follow the label naming pattern used in the code area you are contributing to.

Images, Logo and color

Images and logos used must be subject to Creative Commons CC0.

  • We only use icon from iconmonstr.com.
  • If a color change icon is used, the passive state is gray and the active state is orange.
  • Please add all used images, logos and icons in the Images file.

Format and size

Images, logos and icons are to be created as Portable Network Graphic (PNG) format.

  • The background must be transparent.
  • The basic format is 16x16px.
  • A designation is to be chosen as name. (e.g. information.png)
  • The file name must be written in lower case letters.
  • It must be created in at least two sizes. 16x16px and 32x32px.

Designation basic format as an example:

Color code and status

Color Color code (hex) Color code (RGB) Used for
orange (logo) #f18f01 #f18f01 rgb(241, 143, 1) Activated (ex: filter)
blue (logo) #0e6e8e #0e6e8e rgb(14, 110, 142)
green (logo) #9ac155 #9ac155 rgb(154, 193, 85)
dark blue #95a4b3 #95a4b3 rgb(149, 164, 179) Default color for icons
red #d11d1d #d11d1d rgb(209, 29, 29) Error, Fault
yellow #ffd817 #ffd817 rgb(255, 216, 23) Warning

Interactive-Flex-Query Importer

An XML-compliant document is a file that is structured and formatted according to the rules of the eXtensible Markup Language (XML). It consists of a set of tags and attributes that contain information about the structure and content of the document.

A valid XML document must meet the following conditions:

  • It must have a root element that contains all other elements.
  • Each element must be properly closed.
  • Attributes must be defined within a start tag and their values must be specified within quotes.
  • All elements must be properly nested, i.e. they must contain each other and not overlap.
  • There must be no comments or processing instructions outside the root element.

Source location

Interactive-Flex-Query importer: name.abuchen.portfolio/src/name/abuchen/portfolio/datatransfer/ibflex/

Test cases: name.abuchen.portfolio.tests/src/name/abuchen/portfolio/datatransfer/ibflex/

Imported transactions

Imported transactions are processed in the same way as the PDF importer.

Naming conventions for detected elements

The importer extracts the following elements:

  • type --> The transaction type e.g. buy, sell, deposit, removal, dividends, taxes, fees
  • buySell --> The portfolio transaction type e.g. buy, sell.
  • symbol --> The symbol of the instrument traded.
  • isin --> International Securities Identification Number
  • conid --> The conid of the instrument traded. Same as the security code number (wkn).
  • cusip --> The CUSIP of the instrument traded. Same as the security code number (wkn).
  • quantity --> The number of units for the transaction.
  • multiplier --> The multiplier of the contract traded.
  • proceeds --> Calculated by multiplying the quantity and the transaction price. The proceeds figure will be negative for buys and positive for sales.
  • reportDate --> The date of the statement.
  • tradeDate --> The date of the execution.
  • tradeTime --> The time of the execution.
  • dateTime --> The date and time of the statement.
  • netCash --> Net cash is calculated by adding the proceeds plus tax plus commissions. (buy and sell)
  • amount --> Amount for account transactions
  • currency --> Currency of the total amount.
  • fxRateToBase --> The conversion rate from asset currency to base currency.
  • description --> The description of the instrument traded.
  • taxes --> The total amount of tax for the transaction.
  • ibCommission --> The total amount of commission for the transaction.
  • ibCommissionCurrency --> The currency denomination of the trade.
  • accountId --> Account identification
  • assetCategory --> e.g. Stock, Options, Futures
  • tradeID --> The ID of the trade.
  • transactionID --> The ID of the transaction.
  • accountId --> The account number.

All available elements can be found at ibkrguides.com.

Test cases

The test file is an XML file. People anonymize their personal information and account numbers using alternate text or number strings.

  • The test files should not be modified beyond the anonymization
  • Follow the naming convention for test files (type in the local language, two digit counter):
    • testIBFlexStatementFile01.xml
  • Samples

PDF importer

Importers are created for each supported bank and/or broker. The process works like this:

  • The users selects one or more PDF files via the import menu (or drags and drops multiple PDF files to the sidebar navigation)
  • Each PDF file are converted to an array of strings; one entry per line
  • Each importer is presented with the strings and applies the regular expressions to extract transactions

If you want to add an importer for a new bank or a new transaction type, check out the existing importers for naming conventions, structure, formatting, etc.

Debug information

There is a quick way to generate the required debug information from a pdf file:

File => Import => Debug: Create text from PDF...

Make sure to remove all personal data. However, make also sure that you do not insert any new lines, line breaks, spaces or similar.

Source location

PDF importer: name.abuchen.portfolio/src/name/abuchen/portfolio/datatransfer/pdf/

Test cases: name.abuchen.portfolio.tests/src/name/abuchen/portfolio/datatransfer/pdf/

The naming convention is BANKExtractor and BANKExtractorTest for extractor class and test class respectively.

Imported transactions

Portfolio Performance separates between PortfolioTransaction (booked on a securities account) and AccountTransaction (booked on a cash account). The available types are defined as enum within the file, for example for purchase (BUY) and sale (SELL) of securities, etc.

Anatomy of a PDF importer

The structure of the PDF importers is as follows:

  • Client
    • addBankIdentifier --> Unique recognition feature of the PDF document
  • Bank name
    • getLabel --> display label of bank/broker, e.g. Deutsche Bank Privat- und Geschäftskunden AG
  • Transaction types (basic types)
    • addBuySellTransaction --> Purchase and sale (single settlement)
    • addSummaryStatementBuySellTransaction --> Purchase and sale (multiple settlements)
    • addBuyTransactionFundsSavingsPlan --> Savings plans
    • addDividendeTransaction --> Dividends
    • addTaxTreatmentForDividendeTransaction --> Tax treatment for dividends
    • addAdvanceTaxTransaction --> Advance tax payment
    • addCreditcardStatementTransaction --> Credit card transactions
    • addAccountStatementTransaction --> Giro account transactions
    • addDepotStatementTransaction --> Securities account transactions (settlement account)
    • addTaxStatementTransaction --> Tax settlement
    • addDeliveryInOutBoundTransaction --> Inbound and outbound deliveries
    • addTransferInOutBoundTransaction --> Transfer in and outbound deliveries
    • addReinvestTransaction --> Reinvestment transaction
    • addTaxReturnBlock --> Tax refund
    • addFeeReturnBlock --> Fee refund
  • Taxes and fees
    • addTaxesSectionsTransaction --> handling of taxes
    • addFeesSectionsTransaction --> handling of fees
  • Overwrite the value extractor methods if the documents work with non-standard (English, German) locales:
  • Add post processing on imported transaction using a postProcessing method:

Naming conventions for detected values

The importers are structured according to the following scheme and the mapping variables are to be adhered to as far as possible:

  • Type (Optional)
    • type --> Exchange of the transaction pair (e.g. from purchase to sale)
  • Security identification
    • name --> Security name
    • isin --> International Securities Identification Number
    • wkn --> Security code number
    • tickerSymbol --> Tickersymbol
    • currency --> Security currency
  • Shares of the transaction
    • shares --> Shares
  • Date and time
    • date --> Date
    • time --> Time (optional)
  • Total amount (with fees and taxes)
    • amount --> Amount e.g. 123,15
    • currency --> Currency of the total amount
  • Foreign currency
    • gross --> Total amount in transaction currency without fees and taxes
    • currency --> Currency of the total amount
    • fxGross --> Total amount in foreign currency without fees and taxes
    • fxCurrency --> Currency of the total amount in foreign currency
  • Exchange rate
    • exchangeRate --> Foreign currency exchange rate
    • baseCurrency --> Base currency
    • termCurrency --> Foreign currency
  • Notes (Optional)
    • note --> Notes e.g. quarterly dividend, limits, transaction number
  • Tax section
    • tax --> Amount
    • currency --> Currency
    • withHoldingTax --> Withholding tax
    • creditableWithHoldingTax --> Creditable withholding tax
  • Fee section
    • fee --> Amount
    • currency --> Currency

A finished PDF importer as a basis would be e.g. the V-Bank AG PDF importer.

Auxiliary classes

The utility class about standardized conversions, is called by the AbstractPDFExtractor and processed in the ExtractorUtils. The ExtrExchangeRate helps processing for foreign currencies.

Use the Money class when working with amounts (it includes the currency and the value rounded to cents). Use BigDecimal for exchange rates and the conversion between currencies.

Use TextUtil class for some string manipulation such as trimming strings and stripping whitespace characters. The text created from PDF files has some corner cases that are not supported by the usual Java methods.

Formatting of PDF importer

Due to the many comments with text fragments from the PDF documents, we do not auto-format the PDF importer class files. Instead, carefully insert new code into the existing formatting manually. To protect formatting from automatic formatting, use the @formatter:off and @formatter:on.

Please take a look at the formatting and structure in the other PDF importers! Example: V-Bank AG

Test cases

Via the application menu, users can create a test case file. The test file is the extracted text from the PDF documents. Users then anonymize the text by replacing personal idenfiable information and account numbers with alternative text.

  • The test files should not be modified beyond the anonymization
  • All source code (including the test files) are stored in UTF-8 encoding
  • Follow the naming convention for test files (type in the local language, two digit counter):
    • Buy01.txt, Sell01.txt --> Purchase and sale (single settlements) (e.g. SecurityBuy01.txt or SecuritySale01.txt)
    • Dividend01.txt --> Dividends (single statements)
    • SteuermitteilungDividende01.txt --> Tax settlement for dividends (single settlement)
    • SammelabrechnungKaufVerkauf01.txt --> Purchase and sale (multiple settlements)
    • Wertpapiereingang01.txt --> Incoming securities
    • Wertpapierausgang01.txt --> Outgoing securities
    • Vorabpauschale01.txt --> Advance taxes
    • GiroKontoauzug01.txt --> Giro account statement
    • KreditKontoauszug01.txt --> Credit card account statement
    • Depotauszug01.txt --> security account transaction history (settlement account)
  • Samples
    • From April 2023 we will use the simplified notation of test cases (preferred variant)
    • one transaction per PDF (old version): Erste Bank Gruppe - see testWertpapierKauf06() and testDividende05()
    • supporting securities with multiple currencies: Erste Bank Gruppe with testWertpapierKauf09() / testWertpapierKauf09WithSecurityInEUR() and testDividende10()/testDividende10WithSecurityInEUR()
      • Background: in the PP model, the currency of the transaction always must match the currency of the security and its historical prices. However, sometimes securities are purchased on an different exchange with prices in an another currency. The importer try to handle this case automatically. This is reflected in the two test cases
    • multiple transactions per PDF: DKB AG with testGiroKontoauszug01()
    • if transactions are created based on two separate PDF files, use post processing: Targobank with:
      • testDividende01() (single import)
      • testDividende01WithSecurityInEUR() (single import)
      • testTaxTreatmentForDividende01() (single import)
      • testDividendeWithTaxTreatmentForDividende01() (simultaneously import)
      • testDividendeWithTaxTreatmentForDividende01WithSecurityInEUR() (simultaneously import)
      • testDividendeWithTaxTreatmentForDividende01WithSecurityInUSD() (simultaneously import)
      • testDividendeWithTaxTreatmentReversedForDividende01WithSecurityInEUR() (simultaneously import)
      • testDividendeWithTaxTreatmentReversedForDividende01WithSecurityInUSD() (simultaneously import)

Regular expressions

To test regular expression you can use https://regex101.com/.

Beside general good practices for regular expresions, keep in mind:

  • all special characters in the PDF document (äöüÄÖÜß as well as e.g. circumflex or similar) should be matched by a . (dot) because the PDF to text conversion can create different results
  • the special characters $^{[(|)]}*+?\ in the PDF document are to be escaped
  • expression in .match(" ... ") is started with an anchor ^ and ended with $
  • with .find(" ... ") do not add anchors as they will be automatically added

Keep in mind that the regular expressions work against text that is automatically created from PDF files. Due to the nature of the process, there can always be slight differences in the text files. The following table collects the regular expressions that worked well to match typical values.

Value Example Not Helpful Works Well
Date 01.01.1970 \\d+.\\d+.\\d{4} [\\d]{2}\\.[\\d]{2}\\.[\\d]{4}
1.1.1970 \\d+.\\d+.\\d{4} [\\d]{1,2}\\.[\\d]{1,2}\\.[\\d]{4}
Time 12:01 \\d+:\\d+ [\\d]{2}\\:[\\d]{2}}
ISIN IE00BKM4GZ66 \\w+ [A-Z]{2}[A-Z0-9]{9}[0-9]
WKN A111X9 \\w+ [A-Z0-9]{6}
Valoren 1098758 \\w+ [A-Z0-9]{5,9}
SEDOL B5B74S0 \\w+ [A-Z0-9]{7}
CUSIP 11135F101 \\w+ [A-Z0-9]{9}
Amount 751,68 [\\d,.]+ [\\.,\\d]+
[\\.\\d]+,[\\d]{2}
74'120.00 [\\d.']+ [\\.'\\d]+
20 120.00 [\\d.\\s]+ [\\.\\d\\s]+
Currency EUR \\w+ [A-Z]{3}
[\\w]{3}
Currency € or $ \\D \\p{Sc}

Trade calendar

Using the application menu, the user can select a trading calendar accordingly globally or for each individual security. The calendar takes into account the weekends and regional holidays when there is no stock exchange trading. The trading-free days of the stock exchange itself, if it is a stock exchange calendar, are also taken into account. The individual trading-free days are stored in a HashMap and made available for further processing, e.g. reporting period, performance index and so on.

Source location

Trade calendar Manager: name.abuchen.portfolio/src/name/abuchen/portfolio/util/TradeCalendarManager.java

Trade calendar Class: name.abuchen.portfolio/src/name/abuchen/portfolio/util/TradeCalendar.java

Holiday types: name.abuchen.portfolio/src/name/abuchen/portfolio/util/HolidayType.java

Holiday class: name.abuchen.portfolio/src/name/abuchen/portfolio/util/Holiday.java

Holiday name: name.abuchen.portfolio/src/name/abuchen/portfolio/util/HolidayName.java

Tests: name.abuchen.portfolio.tests/src/name/abuchen/portfolio/util/TradeCalendarTest.java

Anatomy of a trade calendar

The structure of the trade calendars is as follows:

  • Identification
    • code --> Identification nyse --> New York Stock Exchange
  • Label
    • description --> Display label of trade calendar, e.g., New York Stock Exchange
  • Days of weekend
    • weekend --> Sets the default days for the weekend in the Trade Calendar, e.g., Saturday and Sunday

The HolidayTypes helps to edit holidays and in the HolidayName there are all holidays or trade-free days.

Test cases

The test cases are checking individual dates that have been stored in TradeCalendarManager. In these cases we check whether the date to be checked is a trading day for this calendar or not. We assume that every day is a trade day. (Except e.g. regular weekends).

The structure of the test cases is as follows:

  • Starting from the first day of the year with at least three test checks each, e.g. 01.01.20xx, 01.01.20xy, 01.01.20xz until 31.12.20xy of the respective year.
  • The regular trading-free days are followed by the one-time trading-free days.
  • The respective trade-free day to the test is named in the comment.
  • Samples