Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data accuracy tests #17

Merged
merged 26 commits into from
Apr 5, 2022
Merged

Add data accuracy tests #17

merged 26 commits into from
Apr 5, 2022

Conversation

dostrelith678
Copy link
Member

@dostrelith678 dostrelith678 commented Mar 19, 2022

Description

First-take of detailed tests to be run against contributing instances from monitoring, or locally during development/instance verification.
To consider:

  1. Not all endpoints are covered, but all sections are covered (e.g. pool_info tested but pool_blocks not). To be decided if we want to test more or less endpoints (list of tested endpoints included below).
  2. Figure out how exactly is it going to be integrated with monitoring instances to perform these checks against all nodes in the topology. This is to be covered in the next PR, but the current implementation allows specifying the local/compare nodes from the command line (example below).

Endpoints Tested

Network

  • /tip [removed]
  • /genesis
  • /totals

Epoch

  • /epoch_info
  • /epoch_params

Block

  • /block_info
    Transactions
  • /tx_info

Address

  • /address_info

Account

  • /account_info

Asset

  • /asset_info
  • /asset_address_list

Pool

  • /pool_list
  • /pool_info

Script

  • /native_script_list

Where should the reviewer start?

Tests can be executed locally by setting up the test env with setup-tests.py and then running pytest from tests directory. This runs with default params specified in tests/conftest.py. You can pass in the parameters via the command line:
pytest --local-url https://eden-guildnet.koios.rest:8453/api/v0 --compare-url https://guild.koios.rest/api/v0 --api-schema-file ../specs/results/koiosapi-guild.yaml

To specify specific test files:
pytest -s test_totals_endpoint.py

Motivation and context

We need more detailed testing to continually verify both our API schema with calls that are not covered by examples in the API spec (schema changes in this PR are the result of inconsistencies found during testing), and that contributing instances are serving healthy data.

Which issue it fixes?

Closes #9

How has this been tested?

Guild:
image

Testnet:
image

Mainnet:
image

@dostrelith678
Copy link
Member Author

Updated with command line support for specifying test/compare nodes along with the description. Ready for review I think

@dostrelith678 dostrelith678 marked this pull request as ready for review April 4, 2022 17:08
… for http, and update schema (pool ticker is nullable)
Copy link
Contributor

@rdlrt rdlrt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added few cosmetic changes, alongwith example command to use pytest, only thing I'd consider is potentially dropping tip from data tests (since that could go off for seconds difference between instances - on either side), but other than that looks good to merge

tests/test_tip_endpoint.py Outdated Show resolved Hide resolved
Copy link
Contributor

@Scitz0 Scitz0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't touched python much and as such makes it a bit difficult to properly assess code. But I trust in yours and priyanks ability on this.

@dostrelith678 dostrelith678 merged commit a0f19c9 into main Apr 5, 2022
@rdlrt rdlrt deleted the add-data-accuracy-tests branch April 5, 2022 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Test framework..seperate to grest-poll script
3 participants