From b2ca72e6f9e4bc7359d5efe1181d04a40d24a3b2 Mon Sep 17 00:00:00 2001 From: Mark Stacey Date: Wed, 6 Nov 2024 15:20:16 -0330 Subject: [PATCH 1/3] feat: Add testing principles A section has been added to the "Testing Overview" document to describe our testing principles. These establish what the goals of our automated test suite are, and help contributors differentiate between good and bad tests. --- docs/testing/overview.md | 48 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/docs/testing/overview.md b/docs/testing/overview.md index 0b9702c..36aaad6 100644 --- a/docs/testing/overview.md +++ b/docs/testing/overview.md @@ -1,6 +1,52 @@ # Testing Overview -At MetaMask, we are committed to delivering high-quality software. A key aspect of achieving this goal is through a robust testing strategy that includes both end-to-end (e2e), integration and unit tests. This document provides comprehensive guidelines on each testing type, emphasizing the importance of choosing the right test for the right scenario and detailing the specific practices for writing and organizing these tests. +At MetaMask, we are committed to delivering high-quality software. These guidelines describe how we use automated tests to maximize product quality while minimizing the cost of writing and maintaining tests. + +Our automated tests fall into three general categories: end-to-end (e2e), integration and unit tests. This document provides comprehensive guidelines on each testing type, emphasizing the importance of choosing the right test for the right scenario and detailing the specific practices for writing and organizing these tests. + +## Testing principles + +What makes a good test? Here are some general principles. + +### 1. Tests should describe expectations + +What's the difference between a bug and a feature? A feature is expected, a bug is not. + +If it's not clear what the code-under-test is supposed to do, it won't be clear what to do when the test fails either. Clear expectations are essential for an effective test. + +### 2. Tests should catch bugs + +This is the main purpose of tests. Take care to ensure that each test actually fails when the described expectation is not met. + +### 3. Tests should not break often + +It's expected that tests will break when we change the behavior or APIs under test. But if a test breaks often from refactors, or from changes to behavior or APIs not directly under test, it's sign that the test may be poorly designed. + +Test breakages are expensive. We should do what we can to minimize them. + +### 4. Tests should be easy to read + +A test isn't effective if we can't understand why it failed. The more readable a test is, the faster we can fix it. + +### 5. Tests should be easy to write + +Some behaviors are inherently more complex than others, increasing the difficulty of testing them. But in general we want tests to be easy to write, so that we can keep costs low. + +If a test is hard to write, consider that as a warning that you might be approaching the problem in a non-optimal way. Resist the urge to power through the work, and instead consider how redesigning the code-under-test or using a different testing strategy might save time. + +### 6. Tests should be fast + +Making tests faster has a dramatic improvement on productivity, reducing feedback loops. + +What we consider "fast" depends on what the test does; what's fast for an end-to-end test might be slow for a unit test. But faster is always better, for all types of tests. + +### 7. Tests should have a high return on investment + +If a test doesn't align with the previous 6 principles, it's probably doing more harm than good. Consider deleting it. + +That doesn't mean we should skip writing tests with it's too difficult. On the contrary; we have high expectations for quality, we don't have the luxury of writing only the easiest most valuable tests. We need to aim for total coverage. + +But writing and maintaining bad tests is not the shortest path to complete test coverage. Our time is better spent writing good tests, and making investments that allow us to write more good tests. All code can be designed to be easy to test, and even the most complex test setup/teardown can be reasonably fast and easy to read/write with the right tools. ## What are unit tests? From ddd7ad730ecdd8405ff6eb0654dd0315be002203 Mon Sep 17 00:00:00 2001 From: Mark Stacey Date: Thu, 7 Nov 2024 11:59:26 -0330 Subject: [PATCH 2/3] Clarify that test breakages might be caused by poorly designed code, not just poorly designed tests --- docs/testing/overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/overview.md b/docs/testing/overview.md index 36aaad6..698b90e 100644 --- a/docs/testing/overview.md +++ b/docs/testing/overview.md @@ -20,7 +20,7 @@ This is the main purpose of tests. Take care to ensure that each test actually f ### 3. Tests should not break often -It's expected that tests will break when we change the behavior or APIs under test. But if a test breaks often from refactors, or from changes to behavior or APIs not directly under test, it's sign that the test may be poorly designed. +It's expected that tests will break when we change the behavior or APIs under test. But if a test breaks often from refactors, or from changes to behavior or APIs not directly under test, it's sign that the test (or the code-under-test) may be poorly designed. Test breakages are expensive. We should do what we can to minimize them. From 60f5dfd9f3f5160c245a0ce84a31384a496d8eae Mon Sep 17 00:00:00 2001 From: Mark Stacey Date: Fri, 8 Nov 2024 21:47:15 -0330 Subject: [PATCH 3/3] Minor editorial changes --- docs/testing/overview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/testing/overview.md b/docs/testing/overview.md index 698b90e..0c33466 100644 --- a/docs/testing/overview.md +++ b/docs/testing/overview.md @@ -6,13 +6,13 @@ Our automated tests fall into three general categories: end-to-end (e2e), integr ## Testing principles -What makes a good test? Here are some general principles. +The following are general principles about how to write automated tests effectively, applicable to all types of automated tests. ### 1. Tests should describe expectations What's the difference between a bug and a feature? A feature is expected, a bug is not. -If it's not clear what the code-under-test is supposed to do, it won't be clear what to do when the test fails either. Clear expectations are essential for an effective test. +Each test should be verifying specific expectations. If it's not clear what the code-under-test is supposed to do, it won't be clear what to do when the test fails either. Clear expectations are essential for an effective test. ### 2. Tests should catch bugs @@ -44,9 +44,9 @@ What we consider "fast" depends on what the test does; what's fast for an end-to If a test doesn't align with the previous 6 principles, it's probably doing more harm than good. Consider deleting it. -That doesn't mean we should skip writing tests with it's too difficult. On the contrary; we have high expectations for quality, we don't have the luxury of writing only the easiest most valuable tests. We need to aim for total coverage. +That doesn't mean we should skip writing tests when it's too difficult. On the contrary, given our high expectations for quality we don't have the luxury of writing only the easiest tests. We need to aim for total coverage. -But writing and maintaining bad tests is not the shortest path to complete test coverage. Our time is better spent writing good tests, and making investments that allow us to write more good tests. All code can be designed to be easy to test, and even the most complex test setup/teardown can be reasonably fast and easy to read/write with the right tools. +However, writing and maintaining bad tests is not the shortest path to complete test coverage. Our time is better spent writing good tests, and making investments that allow us to write more good tests. All code can be designed to be easy to test, and even the most complex test setup/teardown can be reasonably fast and easy to read/write with the right tools. ## What are unit tests?