Skip to content

Latest commit

 

History

History
1024 lines (830 loc) · 25.2 KB

README.md

File metadata and controls

1024 lines (830 loc) · 25.2 KB
Raw

Google Groups GitHub Discussions

GoCD JSON configuration plugin

GoCD pipeline configuration as code

This is a GoCD server plugin which allows to keep pipelines and environments configuration in version control systems supported by GoCD (git, svn, mercurial, etc.). See this document to find out what GoCD's configuration repositories are.

Table of contents

  1. Setup
  2. File pattern
  3. Validation
  4. Format reference
  5. Format version
  6. Issues and questions
  7. Development
  8. License

Setup

Step 1: GoCD versions newer than 17.8.0 already have the plugin bundled. You don't need to install anything.

If you're using GoCD version older than 17.8.0, you need to install the plugin in the GoCD server. Download it from the releases page and place it on the GoCD server in plugins/external directory.

Step 2: Follow the GoCD documentation to add a new configuration repository.

You can use the example repository at: https://github.com/tomzo/gocd-json-config-example.git as a reference.

In your config repo (tomzo/gocd-json-config-example.git in this case), ensure that your GoCD json config is suffixed with .gopipeline.json for pipelines and .goenvironment.json for environments. Give it a minute or so for the polling to happen. Once that happens, you should see your pipeline(s) on your dashboard.

File pattern

Using this plugin you can store any number of pipeline or environment configurations in a versioned repository like git.

By default pipelines should be stored in *.gopipeline.json files and environments should be stored in *.goenvironment.json files.

The file name pattern can be changed on plugin configuration page.

Validation

You can validate if proposed GoCD JSON changes will be accepted by the server. Currently, 2 options are available:

Validation using CLI

You may find this introductory blog post useful.

There is an ongoing effort to allow in-depth validation of configuration before pushing configuration to the source control. This is provided by GoCD's preflight API and gocd-cli.

You have several options to configure validation tools on your workstation:

Either way you'll have gocd binary in your PATH or inside the docker container.

Syntax validation

This will check general validity of the yaml file, without talking to the GoCD server:

gocd configrepo --json syntax ci.gopipeline.json

Preflight validation

This command will parse and submit your json file to the configured GoCD server.

gocd configrepo preflight --json -r gocd-json-config-example *.gopipeline.json

Where -r is the configuration repository id, which you have earlier configured on GoCD server. You can check it on config repos page of your GoCD server, at /go/admin/config_repos. It is in the upper left corner of each config repo. config repo id

JSON Configuration reference

The pipeline configuration files should be stored in format similar to one exposed by GoCD API.

The format of environment configuration files is much simpler, you can find examples of correct environments below.

  1. Format version
  2. Environment
  3. Environment variables
  4. Pipeline
  5. Stage
  6. Job
  7. Tasks
  8. Materials

Format version

Please note that it is now recommended to declare the same format_version in each *.gopipeline.json or *.goenvironment.json file.

GoCD server version from 19.10.0 and beyond

Supports format_version value of 9. In this version, support of ignore_for_scheduling for dependency materials has been added. Setting this attribute will skip scheduling the pipeline when the dependency material has changed.

Using a newer format_version includes all the behavior of the previous versions too.

GoCD server version from 19.9.0 and beyond

Supports format_version value of 7 and 8. In version 7, support for properties has been removed. In version 8, support for mingle has been removed.

Using a newer format_version includes all the behavior of the previous versions too.

GoCD server version from 19.8.0 and beyond

Supports format_version value of 6. In this version, support of allow_only_on_success for approval on stage has been added. Setting this attribute will ensure that the manual trigger will be allowed only if the previous stage is successful.

Using a newer format_version includes all the behavior of the previous versions too.

GoCD server version from 19.4.0 to 19.7.0

Supports format_version value of 5. In this version, support of username and encrypted_password for git and hg material has been added. In addition to that, hg will also support branch attribute.

Using a newer format_version includes all the behavior of the previous versions too.

GoCD server version from 19.3.0 to 19.4.0

Supports format_version value of 4. In this version, support has been added to control the display order of pipelines.

This server version also supports format_version of 3 and 2. Using a newer format_version includes all the behavior of the previous versions too.

GoCD server version from 18.7.0 to 19.2.0

Supports format_version value of 3. In this version fetch artifact format was changed to include artifact_origin.

This server version also supports format_version of 2. Using a newer format_version includes all the behavior of the previous versions too.

GoCD server version up to 18.6.0

Supports format_version value of 2. In this version pipeline locking behavior was changed.

Environment

Configures a GoCD environment

{
  "name": "dev",
  "environment_variables": [
    {
      "name": "key1",
      "value": "value1"
    }
  ],
  "agents": [
    "123"
  ],
  "pipelines": [
    "mypipeline1"
  ]
}

Environment variables

Environment variables is a JSON array that can be declared in Environments, Pipelines, Stages and Jobs.

Any variable must contain name and value or encrypted_value.

{
  "environment_variables": [
    {
      "name": "key1",
      "value": "value1"
    },
    {
      "name": "keyd",
      "encrypted_value": "v12@SDfwez"
    }
  ]
}

Pipeline

{
  "format_version" : 1,
  "group": "group1",
  "name": "pipe2",
  "label_template": "foo-1.0-${COUNT}",
  "enable_pipeline_locking" : false,
  "parameters": [
    {
      "name": "param",
      "value": "parameter"
    }
  ],
  "tracking_tool": null,
  "timer": {
    "spec": "0 15 10 * * ? *"
  },
  "environment_variables": [],
  "materials": [
    ...
  ],
  "stages": [
    ...
  ]
}

Referencing an existing template in a config repo:

{
  "format_version" : 1,
  "group": "group1",
  "name": "pipe-with-template",
  "label_template": "foo-1.0-${COUNT}",
  "enable_pipeline_locking" : false,
  "template": "template1",
  "parameters": [
    {
      "name": "param",
      "value": "parameter"
    }
  ],
  "materials": [
    ...
  ]
}

Please note:

  • Pipeline declares a group to which it belongs

Pipeline locking

Expected since GoCD v17.12, you need to use lock_behavior rather than enable_pipeline_locking.

"lock_behavior" : "none"

lock_behavior can be one of:

  • lockOnFailure - same as enable_pipeline_locking: true
  • unlockWhenFinished -
  • none - same enable_pipeline_locking: false

Controlling the display order

When format_version is 4 (see above), the order of display of pipelines on the GoCD dashboard can be influenced by setting the display_order_weight property.

  • This is an integer property and the pipelines in a pipeline group will be ordered by this value.
  • The default value for this property is -1.
  • Pipelines defined in GoCD's config XML will also default to -1.
  • If multiple pipelines have the same display_order_weight value, their order relative to each other will be indeterminate.
{
  "name": "pipeline1",
  "group": "pg1",
  "display_order_weight": 10
},
{
  "name": "pipeline2",
  "group": "pg1",
  "display_order_weight": -10
}

In the above example, since both pipelines are in the same group, pipeline2 will be shown ahead of pipeline1. If any pipelines are defined in the GoCD config XML, then they will appear in between these two pipelines.

Timer

{
    "spec": "0 0 22 ? * MON-FRI",
    "only_on_changes": true
}

Tracking tool

{
  "link": "http://your-trackingtool/yourproject/${ID}",
  "regex": "evo-(\\d+)"
}

Mingle

DEPRECATION NOTICE: Since GoCD version 19.9 and format_version 8, this is no longer supported

{
    "base_url": "https://mingle.example.com",
    "project_identifier": "foobar_widgets",
    "mql_grouping_conditions": "status > 'In Dev'"
}

Stage

{
  "name": "test",
  "fetch_materials": true,
  "never_cleanup_artifacts": false,
  "clean_working_directory": false,
  "approval" : null,
  "environment_variables": [
    {
      "name": "TEST_NUM",
      "value": "1"
    }
  ],
  "jobs": [
    ...
  ]
}

Approval

{
  "type": "manual",
  "allow_only_on_success": true,
  "users": [],
  "roles": [
    "manager"
  ]
}

Job

{
  "name": "test",
  "run_instance_count" : null,
  "environment_variables": [],
  "timeout": 180,
  "elastic_profile_id": "docker-big-image-1",
  "tabs": [
     {
       "name": "test",
       "path": "results.xml"
     }
   ],
   "resources": [
    "linux"
  ],
  "artifacts": [
    {
      "source": "src",
      "destination": "dest",
      "type": "test"
    },
    {
      "type": "external",
      "id": "docker-release-candidate",
      "store_id": "dockerhub",
      "configuration": [
        {
          "key": "Image",
          "value": "gocd/gocd-demo"
        },
        {
          "key": "Tag",
          "value": "${GO_PIPELINE_COUNTER}"
        },
        {
          "key": "some_secure_property",
          "encrypted_value": "!@ESsdD323#sdu"
        }
      ]
    }
  ],
  "tasks": [
    ...
  ]
}

Artifacts

There are 3 types of artifacts recognized by GoCD. Build and Test artifacts are stored on the GoCD server. The source and the destination of the artifact that should be stored on the GoCD server must be specified.

Build

{
  "source": "src",
  "destination": "dest",
  "type": "build"
}

Test

{
  "source": "src",
  "destination": "dest",
  "type": "test"
}

External

Artifacts of type external are stored in an artifact store outside of GoCD. The external artifact store's configuration must be created in the main GoCD config. Support for external artifact store config to be checked in as yaml is not available. The external artifact store is referenced by the store_id. The build specific artifact details that the artifact plugin needs to publish the artifact is provided as configuration.

{
  "type": "external",
  "id": "docker-release-candidate",
  "store_id": "dockerhub",
  "configuration": [
    {
      "key": "Image",
      "value": "gocd/gocd-demo"
    },
    {
      "key": "Tag",
      "value": "${GO_PIPELINE_COUNTER}"
    },
    {
      "key": "some_secure_property",
      "encrypted_value": "!@ESsdD323#sdu"
    }
  ]
}

Property

DEPRECATION NOTICE: Since GoCD version 19.9 and format_version 7, properties are no longer supported

{
  "name": "coverage.class",
  "source": "target/emma/coverage.xml",
  "xpath": "substring-before(//report/data/all/coverage[starts-with(@type,'class')]/@value, '%')"
}

Tab

{
      "name": "cobertura",
      "path": "target/site/cobertura/index.html"
}

Run many instances

Part of job object can be number of job to runs

"run_instance_count" : 6

Or to run on all agents

"run_instance_count" : "all"

Default is null which runs just one job.

Materials

All materials:

  • must have type - git, svn, hg, p4, tfs, dependency, package, plugin.
  • can have name and must have name when there is more than one material in pipeline

SCM-related materials have destination field.

Filter - blacklist and whitelist

All scm materials can have filter object:

  • for blacklisting:
"filter": {
  "ignore": [
    "externals",
    "tools"
  ]
}
  • for whitelisting (since Go >=16.7.0):
"filter": {
 "whitelist": [
   "moduleA"
 ]
}

Git

{
  "url": "http://my.git.repository.com",
  "branch": "feature12",
  "filter": {
    "ignore": [
      "externals",
      "tools"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "gitMaterial1",
  "type": "git",
  "shallow_clone": true,
  "username": "user1",
  "encrypted_password": "encrypted_value"
}

For GoCD >= 19.4.0 and format_version: 5 and above:

You are advised to utilize username and encrypted_password for passing in material credentials as:

{
  "url": "http://my.git.repository.com",
  "branch": "feature12",
  "username": "user1",
  "encrypted_password": "encrypted_value"
}
  • Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.
  • Specifying credentials both in attributes and url will result in a validation error e.g. 
      INVALID MERGED CONFIGURATION
  Number of errors: 1+
  1. Ambiguous credentials, must be provided either in URL or as attributes.;;
  - For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0anc

Svn

{
  "url": "http://svn",
  "username": "user1",
  "encrypted_password": "encrypted_value",
  "check_externals": true,
  "filter": {
    "ignore": [
      "tools",
      "lib"
    ]
  },
  "destination": "destDir1",
  "auto_update": false,
  "name": "svnMaterial1",
  "type": "svn"
}

Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.

Hg

{
  "url": "repos/myhg",
  "filter": {
    "ignore": [
      "externals",
      "tools"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "hgMaterial1",
  "type": "hg",
  "username": "user1",
  "encrypted_password": "encrypted_value",
  "branch": "feature"
}

For GoCD >= 19.4.0 and format_version: 5 and above:

You are advised to utilize username and encrypted_password for passing in material credentials as:

{
  "url": "repos/myhg",
  "username": "user1",
  "encrypted_password": "encrypted_value"
}
  • Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.
  • Specifying credentials both in attributes and url will result in a validation error e.g. 
      INVALID MERGED CONFIGURATION
  Number of errors: 1+
  1. Ambiguous credentials, must be provided either in URL or as attributes.;;
  - For Config Repo: https://your.config.repo.url at cbb047d78c239ab23b9565099e800c6fe4cc0anc

In addition to that, you can also leverage branch attribute to specify the branch for material

{
  "branch": "feature"
}

Perforce

{
  "port": "10.18.3.102:1666",
  "username": "user1",
  "encrypted_password": "encrypted_value",
  "use_tickets": false,
  "view": "//depot/dev/src...          //anything/src/...",
  "filter": {
    "ignore": [
      "lib",
      "tools"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "p4materialName",
  "type": "p4"
}

Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.

Tfs

{
  "url": "url3",
  "username": "user4",
  "domain": "example.com",
  "encrypted_password": "encrypted_value",
  "project": "projectDir",
  "filter": {
    "ignore": [
      "tools",
      "externals"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "tfsMaterialName",
  "type": "tfs"
}

Instead of encrypted_password you may specify password but encrypted_password makes more sense considering that the value is stored in SCM.

Dependency

{
  "pipeline": "pipeline2",
  "stage": "build",
  "name": "pipe2",
  "type": "dependency",
  "ignore_for_scheduling": false
}

Package

{
  "package_id": "apt-repo-id",
  "name": "myapt",
  "type": "package"
}

Pluggable SCM

{
  "scm_id": "someScmGitRepositoryId",
  "destination": "destinationDir",
  "filter": {
    "ignore": [
      "dir1",
      "dir2"
    ]
  },
  "name": "myPluggableGit",
  "type": "plugin"
}

Since GoCD >= 19.2.0 defining new pluggable materials that are not defined in the GoCD server is supported.

{
  "plugin_configuration": {
    "id": "plugin_id",
    "version": "1"
  },
  "configuration": [
    {
      "key": "url",
      "value": "[email protected]:tomzo/gocd-json-config-plugin.git"
    }
  ],
  "destination": "destinationDir",
  "filter": {
    "ignore": [
      "dir1",
      "dir2"
    ]
  },
  "name": "myPluggableGit",
  "type": "plugin"
}

Configrepo

This is a convenience for shorter and more consistent material declaration. When configuration repository is the same as one of pipeline materials, then you usually need to repeat definitions in XML and in JSON, for example:

...
  "materials": [
    {
      "url": "https://github.com/tomzo/gocd-json-config-example.git",
      "branch" : "ci",
      "type": "git",
      "name" : "mygit"
    }
  ],
...

And in server XML:

<config-repos>
   <config-repo pluginId="json.config.plugin" id="repo1">
     <git url="https://github.com/tomzo/gocd-json-config-example.git" branch="ci" />
   </config-repo>
</config-repos>

Notice that url and branch is repeated. This is inconvenient in case when you move repository, because it requires 2 updates, in code and in server XML.

Using configrepo material type, above repetition can be avoided, last example can be refactored into:

...
  "materials": [
    {
      "type": "configrepo",
      "name" : "mygit"
    }
  ],
...

Server interprets configrepo material in this way:

Clone the material configuration of the repository we are parsing as is in XML and replace name, destination and filters (whitelist/blacklist), then use the modified clone in place of configrepo material.

Tasks

Every task object must have type field. Which can be exec, ant, nant, rake, fetch, plugin

Optionally any task can have run_if and on_cancel.

  • run_if is a string. Valid values are passed, failed, any
  • on_cancel is a task object. Same rules apply as to tasks described on this page.

Exec

{
    "type": "exec",
    "run_if": "passed",
    "on_cancel" : null,
    "command": "make",
    "arguments": [
      "-j3",
      "docs",
      "install"
    ],
    "working_directory": null
}

Ant

{
  "build_file": "mybuild.xml",
  "target": "compile",
  "type": "ant",
  "run_if": "any",
  "on_cancel" : null,
}

Nant

{
  "type": "nant",
  "run_if": "passed",
  "working_directory": "script/build/123",
  "build_file": null,
  "target": null,
  "nant_path": null
}

Rake

{
  "type": "rake",
  "run_if": "passed",
  "working_directory": "sample-project",
  "build_file": null,
  "target": null
}

Fetch

Fetch artifact from the GoCD server

{
   "type": "fetch",
   "artifact_origin": "gocd",
   "run_if": "any",
   "pipeline": "upstream",
   "stage": "upstream_stage",
   "job": "upstream_job",
   "is_source_a_file": false,
   "source": "result",
   "destination": "test"
 }

Fetch artifact from an external artifact store

{
   "type": "fetch",
   "artifact_origin": "external",
   "run_if": "any",
   "pipeline": "upstream",
   "stage": "upstream_stage",
   "job": "upstream_job",
   "artifact_id": "upstream_external_artifactid",
   "configuration": [
     {
       "key": "DestOnAgent",
       "value": "foo"
     },
     {
       "key": "some_secure_property",
       "encrypted_value": "ssd#%fFS*!Esx"
     }
   ]
 }

Plugin

{
  "type": "plugin",
  "configuration": [
    {
      "key": "ConverterType",
      "value": "jsunit"
    },
    {
      "key": "password",
      "encrypted_value": "ssd#%fFS*!Esx"
    }
  ],
  "run_if": "passed",
  "plugin_configuration": {
    "id": "xunit.converter.task.plugin",
    "version": "1"
  },
  "on_cancel": null
}

Development

Environment setup

To build and test this plugin, you'll need java jdk >= 8.

If you have local java environment, then you may run all tests and create a ready to use jar with:

./gradlew test jar

Building with docker and dojo

You don't need to setup java on your host, if you are fine with using docker and Dojo. This is actually how our GoCD builds the plugin:

dojo "gradle test jar"

Assuming you already have a working docker, On OSX, you can install with homebrew:

brew install kudulab/homebrew-dojo-osx/dojo

A manual install is another option:

version="0.9.0"
# on Linux:
wget -O /tmp/dojo https://github.com/kudulab/dojo/releases/download/${version}/dojo_linux_amd64
# or on Darwin:
# wget -O /tmp/dojo https://github.com/kudulab/dojo/releases/download/${version}/dojo_darwin_amd64
chmod +x /tmp/dojo
mv /tmp/dojo /usr/bin/dojo

Then enter a docker container with java and gradle pre-installed, by running following command at the root of the project:

dojo

Issues and questions

  • If you have questions on usage, please ask them on the GoCD Google Groups forum or GitHub Discussions.

  • If you think there is a bug, or you have an idea for a feature and you are not sure if it's plugin's or GoCD fault/responsibity, please ask on the chat first too.

Please note this brief overview of what is done by the plugin:

  • parsing files into json when GoCD server asks for it.

And this is done by the GoCD server:

  • complex logic merging multiple config repo sources and XML
  • validation of pipelines/stages/jobs/tasks domain
  • any UI rendering

Versioning

We use semantic versioning.

If you are submitting a new feature or patch, please bump the version in build.gradle.

License

Copyright 2019 Tomasz Sętkowski

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.