Skip to content

Code quality guidelines testing and linting

Jump to bottom
Brian Murray edited this page Mar 10, 2022 · 9 revisions

TLDR: We expect integration tests for single operation examples and scenario examples, and unit tests for cross-service examples. All submitted code is automatically linted, and the assigned reviewer is responsible for checking that all code has test sufficient that the code runs as expected.

This document explains:

Terminology in this document

  • Mocking - is synthesizing an input to test that the example runs as it is supposed to. Most languages have mocking frameworks.
  • Unit test - individual modules of an application in isolation (without any interaction with dependencies) to confirm that the code is doing things right. This is the most basic, cheapest form of testing. They require mocking, which is removing the dependencies your system has and replacing it with an implementation you control (for example if your code has external dependencies like system calls, or accessing a database).
  • Integration test - modules of code to verify how systems interact with each other. They can confirm endpoints are live and how we use them is correct. This is a more expensive and thorough form of testing than unit testing. For integration tests you should only test what you can control/edit, and mock anything you can’t.
  • Linting - the automated checking of your source code for programmatic and stylistic errors. See the Linters run on check in for information on your languages’ recommended linting tools.
  • Unit testing framework - software tools to support writing and running unit tests. See Recommended Unit test frameworks for information on your languages’ recommended unit testing framework.

HOW we test code examples

Single operation examples

Generally, we recommend you don’t test single-operation examples unless you’re processing some data, or the single operation example is rare. For example, if the single operation is covered in an MVP scenario, the integration test for the scenario covers it. One exception is we recommend you have a least one single-operation integration test using a unit test framework to demonstrate to users how they can create tests. Single operation API calls which aren’t covered by scenarios should still have simple tests to prove they work. These examples tend to fall on one of two ends of a spectrum: trivially simple calls and obscure/complex calls. For complex calls, the example is useful to end users even if we haven’t found a scenario to use it in. For the simple calls, they’re also trivial to create, so there’s little reason to not include them. The only caveat with single operation example integration tests is to ensure any calls which create resources also have matching teardown calls to accompany the tests.

Scenario examples

We recommend using integration tests that test the relationship between services in your example, using mocking for those you cannot edit/control.

Integration tests rely on external code and test that any relationships to dependencies are set up correctly. Because of this, integration tests should be written and run with caution, as they may incur costs. This is especially true in case they fail and leave resources running that might be racking up charges. Integration test insure that your code properly “integrates” with specific dependencies. It is usually beneficial to test as few dependencies at a time as possible. This reduces scope of each test and reduces the chance that one dependency will give a false negative result because of the failure of a separate process.

Example 1 This PHP S3 test example simply runs the scenario as if it were a user. Because the scenario was designed with testing in mind and runs completely automatically (as long as credentials are setup correctly), this is the most simple form of creating a Scenario test.

Example 2 This Python S3 test example does a similar job to the PHP one, except it also simulates user input. This way, you can have a more interesting interactive scenario while keeping the automation concise.

Example 3 This Rust S3 test example runs against a service library instead of running the scenario example raw. All of the actions run in the scenario are tested independently here. This gives more granularity and insight into where failures occur. Instead of getting an error that a scenario failed to run, specific actions from the service library will fail or succeed. This reduces the cognitive load when investigating failures. It is the best type of integration test to use on larger segments of code that have well defined actions.

Cross-service examples

Unit testing elements of cross-service examples is important to verify that your business logic is correct. This sort of test isolates the rest of the world away, creating a bubble for your tested code to run in.

A few examples of places where unit tests are important:

  • A function that counts the number of words in a report.
  • A function that finds words that contain the given letters (like a scrabble searcher).
  • Text formatting.
  • Parsing the EXIF data out of a photo (Note: while most people would use an external library, it’s still important to verify you’re accessing the EXIF data correctly once you have it.)

Most of these examples fall into the category of “sorting and searching data”.

A less obvious form of this is parsing a structure. In the wordfreq example in gov2, there is a small helper that handles the kinds of messages that an SQS queue might have.

Example 1 This Go unit test example tests this Go example, which helps fill in several structures, each possible forms of the kind of Amazon Simple Queue Service (Amazon SQS) message that could come to the app. The unit test makes sure that for given known good samples of message, the messages are identified and broken up appropriately.

WHY we test code examples

  • To demonstrate to internal stakeholders that our code works and they can feel safe including it in their documentation and recommending it to customers.
  • To reduce and ideally eliminate instances of publishing code that doesn’t work.
  • To ensure our examples won’t stop working without us noticing.
  • To demonstrate to our customers how they can test examples that use AWS services.

WHERE we test code examples

  • We add tests to our code examples before we publish them. There are different testing requirements for each of single-operation, scenario, and cross-service examples.
  • On GitHub PR checkin we run a language-specific linters against all code to ensure it compiles/passes against newly submitted code.
  • (Stretch goal) On GitHub PR checkin we run all unit tests for all languages.
  • (Stretch goal)We schedule and run all tests, including integration tests, on a regular cadence (daily, weekly, etc. to be determined).

WHOES code examples are tested

Code without tests is not merged to this repo. However, if you cannot provide tests, the assigned reviewer will work with you to resolve the issue. At a mimimum all submitted code should be linted.

WHEN we test code examples

Currently, the assigned reviewer is responsible for checking that all code has test sufficient that the code runs as expected.

Eventually, out goal is to have all unit tests, new integration tests, and linters on new code run whenever we create a new PR into main. This will ensure that all code moving forward adheres to our standards. In addition, we have another goal of having all tests run on a weekly basis to ensure the code is up-to-date with underlying dependancies.

Linters run on check in

Here are the lint tools run against code submitted to the repo.

Language Unit test tool License used
CLI shUnit2 Apache License 2.0 - OK
C++ CTest (included with CMake)
C# xUnit, Moq Apache License 2.0
DotNet
Go go test
Java Junit5
JavaScript Jest
PHP PHPUnit
PowerShell Pester
Python pytest MIT license
Ruby RSpec
Rust cargo test

Recommended Unit test frameworks

Language Unit test tool License used
CLI shUnit2 Apache License 2.0 - OK
C++ CTest (included with CMake)
C# xUnit, Moq Apache License 2.0
DotNet
Go go test
Java Junit5
JavaScript Jest
PHP PHPUnit
PowerShell Pester
Python pytest MIT license
Ruby RSpec
Rust cargo test
Clone this wiki locally