Keeping track of what versions are currently deployed using record-deployment

[bethesque]

Working out how to keep track of what is currently deployed has turned out to require more thought than I expected. Recording a deployment is easy, but making the logic resilient to duplicate calls (when a deploy script gets re-run), and also working out what to mark as undeployed (without having to run an explicit record undeployment command each time) is a bit harder. I had already implemented a --replace-previous-deployed-version flag on the CLI, but it turned out to be inadequate for the scenarios I worked through, and I will be replacing it with another mechanism which will be explained below.

There are two main use cases, and one less common, but more complicated one.

1. A single application instance exists in an environment - eg. an API provider running on a server/cluster

• Conceptually, this is a single "instance" of the application in production, no matter how many servers there are.
• Whenever a new deployment is recorded, any previously deployed versions with the same application/environment should be marked as undeployed.
• Duplicate calls for the same app/version/environment should not result in a duplicate "deployed version" being created in the broker

2. A mobile or desktop application being released to a distribution platform

• Conceptually, there are multiple instances of the application in production, one for each version released
• Whenever a new deployment is recorded, the previously released versions for the same application/environment should NOT be marked as undeployed.
• If an application version is deemed to be unsupported, a manual record undeployment call can be run for the unsupported version.
• Duplicate calls for the same app/version/environment should not result in a duplicate "deployed version" being created in the broker.

3. (Complicated case) The situation where multiple (but a finite, known number of) instances of an application are running in production for a non-trivial amount of time (ie. not just just a few minutes during a no-downtime deployment)

• Examples
  • deployments where the traffic is switched over gradually from one server to another over the course of a number hours
  • situations where an organisation manages a consumer application that has a known number of instances that might have different versions in them, but that share some common provider, and so conceptually shares the same "production" environment.
    • eg. a separate managed instance for each client
    • eg. separate instances in different cloud computing regions (for privacy/legislation compliance reasons)
• In this case, whenever a new deployment is recorded, the previously released versions for the same application/environment should NOT automatically be marked as undeployed.
• Duplicate calls for the same app/version/environment potentially SHOULD result in a duplicate "deployed version" being created in the broker.
• PROBLEM: Working out which version to undeploy would involve keeping track of which version was being deployed over for a particular instance, and then marking that version as undeployed, but only if there were no longer any other instances that had that particular version (something that the broker doesn't currently know), so require manual intervention.
• Having already implemented the record deployment API so that it allowed duplicate app/version/environment rows, it got very messy working out what the "record undeployment" command should do if there were multiple rows (undeploy all, or first or last, or allow the user to specify, but what if they didn't know there were multiples, and what if the multiple was there unintentionally...). So I decided that the record deployment API should only ever allow one record for an app/version/environment combination, but then I realised it wouldn't cope properly with this "multiple known instances in prod" situation if I did that.

Solution

All of these situations and the problems associated with keeping track of what is and isn't currently in production can be solved by

• adding an extra column to the "currently deployed version" object - a "logical instance identifier"
• when a new deployment is recorded matching an existing identifier/application/environment combination, the pre-existing one gets marked as undeployed because we know the most recently version has been "deployed over" that one, making the transaction atomic.

Scenarios:

• For the single application instance situation, the instance name would be null. Therefore, only one row would be allowed for each app/env. Each time a deployment is recorded, the previous row for the app/env is marked as undeployed. 👍🏽
• For the mobile consumer situation, the instance name would be the app version number, so you can get one currently deployed version per version. Each time a deployment is recorded, a new row is created, so nothing gets marked as undeployed.👍🏽
• For the multiple but known number of instances, the instance name would be specified by the user at deploy time, so there would be one row per app/environment/[client/region/whatever]. Each time a deployment takes place for an existing instance/application/environment, any previous row with that combination is marked as undeployed 👍🏽

The technical detail of all this would be hidden from the user, and they would get more user friendly CLI parameters. I'm having trouble thinking of good names for them. These are the best I've got so far, but "instance" is an overloaded term. Would love to hear some other suggestions.

pact-broker record-deployment --pacticipant Foo --version 123 --environment prod --single-application-instance
pact-broker record-deployment --pacticipant Foo --version 123 --environment prod --multiple-application-instances
pact-broker record-deployment --pacticipant Foo --version 123 --environment prod --multiple-application-instances --instance mycustomer1

I guess I could actually keep the existing --replace-previous-deployed-version parameter and just add an instance param. Those scenarios would look like:

pact-broker record-deployment --pacticipant Foo --version 123 --environment prod --replace-previous-deployed-version
pact-broker record-deployment --pacticipant Foo --version 123 --environment prod --no-replace-previous-deployed-version
pact-broker record-deployment --pacticipant Foo --version 123 --environment prod --replace-previous-deployed-version --instance mycustomer1

[abubics]

Rather than --[single|multiple]-application-instance[s] maybe like --[single|multiple]-live-version[s] would be intuitive?
It runs a risk of implying that the identifier is the version number (which may be often, but not always, true).

[bethesque]

Maybe "deployment target" would be better than "instance".

[bethesque]

"Deployment disambiguator" 😂

[bethesque]

I had made a proposal for providing a "record release" (as distinct from "record deployment") for library/mobile client releases. Maybe using that would help make a more easily understood CLI.

pact-broker record-deployment --pacticipant foo --version 1 --environment prod 
pact-broker record-release --pacticipant foo --version 1 --environment prod 
pact-broker record-deployment --pacticipant foo --version 1 --environment prod --target mycustomer1

[bethesque]

Ok, I have decided that this is the plan ^^. I'm going to model releases and deployments differently, and that gets rid of the whole multiple/single/replace-previous language.

• A deployment is when you put a particular version of an application into a particular environment, and later deployments cause the previously deployed thing to be undeployed. An optional "target" will allow deployments to be disambiguated when there are multiple application instances in the same environment.
• A release is when you make a particular version of an application available in a particular environment, but later releases do not make the previously released thing un-released. Releases do not have the concept of a target. It is not needed because the purpose of the target is to allow us to know what version was undeployed when we record a new deployment, and a release doesn't have that concept.

Question - does it make sense to release to a non-production environment? I think not. For a mobile app, as far as I understand it, you would record-deployment to a device within a test environment, but do a record-release to production.

[bethesque]

@TimothyJones does this sound sane to you?

[TimothyJones]

Yes! I like it.

I think it’s technically possible to release to an environment (eg multiple artefactories), but probably not sensible/helpful for us to model that case.

[bethesque]

Yeah - the main difference still being that releasing to one repository does not un-release it from any other the others.