Making it easier to get verification results from test/prod provider versions when contracts change

[bethesque]

Background

To understand whether or not we can deploy a consumer when its contract has changed, we need to get verification results from:

• the head version of the provider from its main branch (the workflow for this is described in Step 4 of the CI/CD setup guide)
• the versions of the provider deployed to the test and production environments (the workflow for this is described in Step 8 of the CI/CD setup guide, but it only talks about the production environment, not the test ones.)

If the main branch verification passes and the test/prod verifications pass, we know we can deploy straight to production (can-i-deploy will be successful for each environment).

If the main branch verification passes, but the test and/or production versions fail, we know we need to wait for the latest version of the provider to be deployed into test/prod (can-i-deploy will block us from deploying to any failed environments).

If main/test/prod all fail, then we know that the provider still needs to implement the required changes (can-i-deploy will block us from deploying to any environment).

If the main branch passes, but we don't have any test/prod verification results, then we might actually have a compatible consumer, but can-i-deploy will stop us from deploying, because we don't have a successful verification result from any of the deployed versions.

The problem

Getting all the required verification results (main/test/prod) required to deploy is currently difficult to do efficiently (ie. without excess unnecessary builds) and requires a high level of expertise in the Pact Broker usage. Currently, only the main branch verification result is well documented and easy to implement, meaning that many can-i-deploy queries will be coming back false (saying "no verification found") even if the currently deployed provider does actually implement the contract. When this happens, the consumer needs to wait unnecessarily for the latest version of the provider to be deployed, slowing the releases down.

Proposed solution

Soon, the Pact Broker will have:

• support for specifying the name of the main branch as a property of the pacticipant
• support for specifying the branch with each version published
• support for recording deployments/releases.

This means that when a contract is published, the Pact Broker will be able to determine whether there is a verification result missing from any of the critical provider versions (latest main/current test/current prod).

A new event would be added called something like contract_content_requiring_verification_published. The user would need to create a webhook for this event that triggered a build that took the version number as a parameter (eg. ${pactbroker.targetProviderVersion}). When the broker detected that content had been published that did not have a verification result from any of the critical provider versions, it would trigger the webhook with the following combination of parameters.

• If there was no verification result from the latest known version of the main branch of the provider, trigger the webhook with ${pactbroker.targetProviderVersion} set to the latest version from the main branch.
• For each deployed/released version of the provider, if there is no verification result from that version, trigger a webhook with the version set to the values from that deployed version.
• If any of latest main/test/prod are actually the same version, we can identify this, and only trigger one build for each version.

Prerequisites:

• the branch is published with all pacts/verification results
• the deployments/releases are recorded using the record-deployment/record-release APIs.
• the value of the version number that was published to the Pact Broker IS the git sha (as recommended) or the sha can be reverse engineered by the triggered build, so that it can check out the correct version of the provider. (Side thought - do we want to make an extra field on the version resource for the git sha to support this, so that people who publish pacts with something that isn't the git sha can still use this easily?)

Scenarios

Brand new feature, provider implements change on feature branch (WIP enabled)

• Consumer makes feat/x branch, changes contract, pushes, publishes pact.
  • latest main = no verification, test = no verification, prod = no verification
• Broker triggers builds for latest master, test, prod versions
  • latest main = failed, test = failed, prod = failed
• Consumer commits on feature branch, but does not change pact, pushes, pact published
  • latest main = failed, test = failed, prod = failed
• Broker detects that there are results from main, test and prod, so does not trigger anything
• Provider commits on main, verifies feature pact as WIP, does not do anything in relation to pacts, publishes a new version
  • latest main = still failed
• Consumer commits on feature branch, but does not change pact, pushes, pact published
  • latest main = still failed, test = failed, prod = failed
• Broker detects that is a result from the latest main, because it was verified as WIP, so does not trigger a build.
• Provider makes a feature branch, implements the changes, pushes, publishes successful verification result from the branch
  • latest main = still failed, test = failed, prod = failed
• Provider merges branch into master, verifies feature pact as WIP, publishes successful verification result from main branch
  • latest main = successful, test = failed, prod = failed
• Consumer commits on feature branch, but does not change pact, pushes, pact published
  • latest main = successful, test = failed, prod = failed
• Broker does not trigger any builds
• Consumer merges branch into main
  • latest main = successful, test = failed, prod = failed
• Provider deploys to test
  • latest main = successful, test = successful, prod = failed
• Consumer deploys to test
• Provider deploys to prod
  • latest main = successful, test = successful, prod = successful
• Consumer deploys to prod

[bethesque]

@pendsley would love your thoughts on this