Added format_version 10, move away from blacklist/whitelist

Old formats are forwards compatible and new version is backwards compatible to enable smooth migration for users. See also gocd/gocd#8266
adnovum · Sep 10, 2020 · 935d0bc · 935d0bc
1 parent c8942b7
commit 935d0bc
Show file tree

Hide file tree

Showing 30 changed files with 428 additions and 108 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,6 @@
 ### 0.13.0 - Unreleased
 
-* whitelist to includes rename
+* whitelist to includes rename, blacklist to ignore
 * V3 capabilities API implementation
 * Support `includes` in the JSON input for pipeline export. Related to: https://github.com/gocd/gocd/pull/8266
 * Fix for pipeline export with multiple materials without name, \#144

diff --git a/README.md b/README.md
@@ -222,19 +222,27 @@ Feel free to improve it!
 
 Please note that it is now recommended to declare `format_version` in each `gocd.yaml` file, consistent across all your files.
 
-#### GoCD server version from 19.10.0 and beyond
+#### GoCD server version from 20.8.0 and beyond
 
-Supports `format_version` value of `9`. In this version, support of `ignore_for_scheduling` for [dependency materials](#dependency) has been added. Setting this attribute will skip scheduling the pipeline when the dependency material has changed.
+Supports `format_version` value of `10`. This version removes usage of `blacklist/whitelist` keywords in favour of `includes/ignore`. This is done in a backwards compatible way - you can upgrade the server and plugin first, and update the format_version to 10 after. Just replace `whitelist` by `includes`, and `blacklist` by `ignore`.
+
+Additionally, plugin is also "forwards compatible" since version 0.13.0. If you can't upgrade GoCD to 20.8.0, but want to migrate from `blacklist/whitelist` to `includes/ignore`, you can upgrade the plugin and keep the format_version that you are using. Plugin will migrate keywords under the hood, while your YAML can use `includes/ignore`.
 
 Using a newer `format_version` includes all the behavior of the previous versions too.
 
 ```yaml
-format_version: 9
+format_version: 10
 pipelines:
   ...
 environments:
 ```
 
+#### GoCD server version from 19.10.0 and beyond
+
+Supports `format_version` value of `9`. In this version, support of `ignore_for_scheduling` for [dependency materials](#dependency) has been added. Setting this attribute will skip scheduling the pipeline when the dependency material has changed.
+
+Using a newer `format_version` includes all the behavior of the previous versions too.
+
 #### GoCD server version from 19.9.0 and beyond
 
 Supports `format_version` value of `7` and `8`. In version `7`, support for [properties](#property) has been removed. In version `8`, support for `mingle` as a [tracking tool](#tracking-tool) has been removed.
@@ -248,75 +256,33 @@ Supports `format_version` value of `6`. In this version, support of `allow_only_
 
 Using a newer `format_version` includes all the behavior of the previous versions too.
 
-```yaml
-format_version: 6
-pipelines:
-  ...
-environments:
-```
-
 #### GoCD server version from 19.4.0 to 19.7.0
 
 Supports `format_version` value of `5`. In this version, support of `username` and `encrypted_password` for [git](#git-material-update) and [hg](#hg-material-update) material has been added. In addition to that, [hg](#hg-material-update) will also support `branch` attribute.
 
 Using a newer `format_version` includes all the behavior of the previous versions too.
 
-```yaml
-format_version: 9
-pipelines:
-  ...
-environments:
-```
-
 
 #### GoCD server version 19.3.0
 
 Supports `format_version` value of `4`. In this version, support has been added to control the [display order of pipelines](#display-order-of-pipelines).
 
 This server version also supports `format_version` of `3` and `2`. Using a newer `format_version` includes all the behavior of the previous versions too.
 
-```yaml
-format_version: 4
-pipelines:
-  ...
-environments:
-```
-
 #### GoCD server version from 18.7.0 to 19.2.0
 
 Supports `format_version` value of `3`. In this version [fetch artifact](#fetch) format was changed to include `artifact_origin`.
 
 This server version also supports `format_version` of `2`. Using a newer `format_version` includes all the behavior of the previous versions too.
 
-```yaml
-format_version: 3
-pipelines:
-  ...
-environments:
-```
-
 #### GoCD server version from 17.12.0 to 18.6.0
 
 Supports `format_version` value of `2`. In this version [pipeline locking](#pipeline-locking) behavior was changed.
 
-```yaml
-format_version: 2
-pipelines:
-  ...
-environments:
-```
-
 #### GoCD server version up to 17.11.0
 
 Supports `format_version` value of `1`. This is the initial version.
 
-```yaml
-format_version: 1
-pipelines:
-  ...
-environments:
-```
-
 # Pipeline
 
 A minimal [pipeline](https://docs.gocd.org/current/configuration/configuration_reference.html#pipeline) configuration must contain:
@@ -680,6 +646,10 @@ More customized git material is possible:
 gitMaterial1:
   git: "http://my.git.repository.com"
   branch: feature12
+  ignore:
+    - externals/**/*.*
+    - tools/**/*.*
+# For GoCD < 20.8.0 or plugin version < 0.13.0, you need to use blacklist instead of ignore:
   blacklist:
     - externals/**/*.*
     - tools/**/*.*
@@ -688,12 +658,15 @@ gitMaterial1:
   shallow_clone: true
 ```
 
-Since GoCD `>= 16.7.0` whitelist is also supported,
-you can specify `whitelist` **instead** of `blacklist`, as such
+Since GoCD `>= 16.7.0` includes is also supported,
+you can specify `includes` **instead** of `ignore`, as such
 ```yaml
 gitMaterial1:
   git: "[email protected]"
   branch: "feature12"
+  includes:
+    - src/**/*.*
+# For GoCD < 20.8.0 or plugin version < 0.13.0, you need to use whitelist instead:
   whitelist:
     - src/**/*.*
 ```
@@ -729,6 +702,10 @@ svnMaterial1:
   username: "user1"
   encrypted_password: "encrypted_value"
   check_externals: true
+  ignore:
+    - tools
+    - lib
+# For GoCD < 20.8.0 or plugin version < 0.13.0, you need to use blacklist instead of ignore:
   blacklist:
     - tools
     - lib
@@ -742,6 +719,10 @@ Instead of `encrypted_password` you can specify `password`.
 ```yaml
 hgMaterial1:
   hg: repos/myhg
+  ignore:
+    - externals
+    - tools
+# For GoCD < 20.8.0 or plugin version < 0.13.0, you need to use blacklist instead of ignore:
   blacklist:
     - externals
     - tools
@@ -792,6 +773,10 @@ p4Material1:
   view: |
     //depot/external... //ws/external...
     //depot/tools... //ws/external...
+  ignore:
+    - externals
+    - tools
+# For GoCD < 20.8.0 or plugin version < 0.13.0, you need to use blacklist instead of ignore:
   blacklist:
     - externals
     - tools
@@ -810,6 +795,10 @@ Instead of `encrypted_password` you can specify `password`.
 myPluggableGit:
   scm: someScmGitRepositoryId
   destination: destinationDir
+  ignore:
+    - dir1
+    - dir2
+# For GoCD < 20.8.0 or plugin version < 0.13.0, you need to use blacklist instead of ignore:
   blacklist:
     - dir1
     - dir2
@@ -818,7 +807,7 @@ myPluggableGit:
 Since GoCD `>= 19.2.0` defining new pluggable materials that are not defined
 in the GoCD server is supported.
 
-```
+```yaml
 myPluggableGit:
   plugin_configuration:
     id: plugin_Id
@@ -828,6 +817,10 @@ myPluggableGit:
   secure_options:
     password: "encrypted_value"
   destination: destinationDir
+  ignore:
+    - dir1
+    - dir2
+# For GoCD < 20.8.0 or plugin version < 0.13.0, you need to use blacklist instead of ignore:
   blacklist:
     - dir1
     - dir2
@@ -869,7 +862,7 @@ materials:
 
 Server interprets `configrepo` material in this way:
 
-> Clone the material configuration of the repository we are parsing **as is in XML** and replace **name, destination and filters (whitelist/blacklist)**,
+> Clone the material configuration of the repository we are parsing **as is in XML** and replace **name, destination and filters (includes/ignore)**,
 then use the modified clone in place of `configrepo` material.
 
 ### Dependency

diff --git a/src/main/java/cd/go/plugin/config/yaml/transforms/MaterialTransform.java b/src/main/java/cd/go/plugin/config/yaml/transforms/MaterialTransform.java
@@ -28,7 +28,6 @@ public class MaterialTransform extends ConfigurationTransform {
     public static final String YAML_MATERIAL_IGNORE_FOR_SCHEDULING_FIELD = "ignore_for_scheduling";
 
     public static final String YAML_SHORT_KEYWORD_GIT = "git";
-    //TODO others
 
     public static final String YAML_BLACKLIST_KEYWORD = "blacklist";
     private static final String YAML_SHORT_KEYWORD_DEPENDENCY = "pipeline";
@@ -48,13 +47,14 @@ public class MaterialTransform extends ConfigurationTransform {
 
     public MaterialTransform() {
         yamlSpecialKeywords.add(YAML_SHORT_KEYWORD_GIT);
-        // TODO all other transforms
         yamlSpecialKeywords.add("name");
         yamlSpecialKeywords.add("type");
         yamlSpecialKeywords.add("auto_update");
         yamlSpecialKeywords.add("shallow_clone");
         yamlSpecialKeywords.add("blacklist");
         yamlSpecialKeywords.add("whitelist");
+        yamlSpecialKeywords.add("includes");
+        yamlSpecialKeywords.add("excludes");
         yamlSpecialKeywords.add("scm_id");
         yamlSpecialKeywords.add("package_id");
         yamlSpecialKeywords.add("svn");
@@ -67,10 +67,10 @@ public MaterialTransform() {
         yamlSpecialKeywords.add(YAML_MATERIAL_IGNORE_FOR_SCHEDULING_FIELD);
     }
 
-    public JsonObject transform(Object maybeMaterial) {
+    public JsonObject transform(Object maybeMaterial, int formatVersion) {
         Map<String, Object> map = (Map<String, Object>) maybeMaterial;
         for (Map.Entry<String, Object> entry : map.entrySet()) {
-            return transform(entry);
+            return transform(entry, formatVersion);
         }
         throw new RuntimeException("expected material hash to have 1 item");
     }
@@ -140,7 +140,7 @@ public Map<String, Object> inverseTransform(Map<String, Object> material) {
         return inverseMaterial;
     }
 
-    public JsonObject transform(Map.Entry<String, Object> entry) {
+    public JsonObject transform(Map.Entry<String, Object> entry, int formatVersion) {
         String materialName = entry.getKey();
         JsonObject material = new JsonObject();
         material.addProperty(JSON_MATERIAL_NAME_FIELD, materialName);
@@ -153,8 +153,13 @@ public JsonObject transform(Map.Entry<String, Object> entry) {
         addOptionalBoolean(material, materialMap, JSON_MATERIAL_IGNORE_FOR_SCHEDULING_FIELD, YAML_MATERIAL_IGNORE_FOR_SCHEDULING_FIELD);
         if (materialMap.containsKey("blacklist"))
             addFilter(material, materialMap.get("blacklist"), "ignore");
+        if (materialMap.containsKey("ignore"))
+            addFilter(material, materialMap.get("ignore"), "ignore");
+        String jsonIncludesKeyword = formatVersion < 10 ? "whitelist" : "includes";
+        if (materialMap.containsKey("includes"))
+            addFilter(material, materialMap.get("includes"), jsonIncludesKeyword);
         if (materialMap.containsKey("whitelist"))
-            addFilter(material, materialMap.get("whitelist"), "includes");
+            addFilter(material, materialMap.get("whitelist"), jsonIncludesKeyword);
 
         String git = getOptionalString(materialMap, YAML_SHORT_KEYWORD_GIT);
         if (git != null) {
@@ -207,11 +212,11 @@ public JsonObject transform(Map.Entry<String, Object> entry) {
     private void addInverseFilter(Map<String, Object> material, Map<String, Object> filterList) {
         List<String> filter = (List<String>) filterList.get("ignore");
         if (filter != null && !filter.isEmpty()) {
-            material.put("blacklist", filter);
+            material.put("ignore", filter);
         }
         filter = (List<String>) filterList.get("includes");
         if (filter != null && !filter.isEmpty()) {
-            material.put("whitelist", filter);
+            material.put("includes", filter);
         }
     }
 

diff --git a/src/main/java/cd/go/plugin/config/yaml/transforms/PipelineTransform.java b/src/main/java/cd/go/plugin/config/yaml/transforms/PipelineTransform.java
@@ -51,15 +51,15 @@ public PipelineTransform(MaterialTransform materialTransform, StageTransform sta
         this.parameterTransform = parameterTransform;
     }
 
-    public JsonObject transform(Object maybePipe) {
+    public JsonObject transform(Object maybePipe, int formatVersion) {
         Map<String, Object> map = (Map<String, Object>) maybePipe;
         for (Map.Entry<String, Object> entry : map.entrySet()) {
-            return transform(entry);
+            return transform(entry, formatVersion);
         }
         throw new RuntimeException("expected pipeline hash to have 1 item");
     }
 
-    public JsonObject transform(Map.Entry<String, Object> entry) {
+    public JsonObject transform(Map.Entry<String, Object> entry, int formatVersion) {
         String pipelineName = entry.getKey();
         JsonObject pipeline = new JsonObject();
         pipeline.addProperty(JSON_PIPELINE_NAME_FIELD, pipelineName);
@@ -79,7 +79,7 @@ public JsonObject transform(Map.Entry<String, Object> entry) {
         if (jsonEnvVariables != null && jsonEnvVariables.size() > 0)
             pipeline.add(JSON_ENV_VAR_FIELD, jsonEnvVariables);
 
-        addMaterials(pipeline, pipeMap);
+        addMaterials(pipeline, pipeMap, formatVersion);
         if (!pipeline.has(JSON_PIPELINE_TEMPLATE_FIELD)) {
             addStages(pipeline, pipeMap);
         }
@@ -189,19 +189,19 @@ private JsonArray transformStages(List<Object> stagesList) {
         return stagesArray;
     }
 
-    private void addMaterials(JsonObject pipeline, Map<String, Object> pipeMap) {
+    private void addMaterials(JsonObject pipeline, Map<String, Object> pipeMap, int formatVersion) {
         Object materials = pipeMap.get(YAML_PIPELINE_MATERIALS_FIELD);
         if (!(materials instanceof Map))
             throw new YamlConfigException("expected a hash of pipeline materials");
         Map<String, Object> materialsMap = (Map<String, Object>) materials;
-        JsonArray materialsArray = transformMaterials(materialsMap);
+        JsonArray materialsArray = transformMaterials(materialsMap, formatVersion);
         pipeline.add(JSON_PIPELINE_MATERIALS_FIELD, materialsArray);
     }
 
-    private JsonArray transformMaterials(Map<String, Object> materialsMap) {
+    private JsonArray transformMaterials(Map<String, Object> materialsMap, int formatVersion) {
         JsonArray materialsArray = new JsonArray();
         for (Map.Entry<String, Object> entry : materialsMap.entrySet()) {
-            materialsArray.add(materialTransform.transform(entry));
+            materialsArray.add(materialTransform.transform(entry, formatVersion));
         }
         return materialsArray;
     }

diff --git a/src/main/java/cd/go/plugin/config/yaml/transforms/RootTransform.java b/src/main/java/cd/go/plugin/config/yaml/transforms/RootTransform.java
@@ -30,22 +30,30 @@ public RootTransform(PipelineTransform pipelineTransform, EnvironmentsTransform
 
     public String inverseTransformPipeline(Map<String, Object> pipeline) {
         Map<String, Object> result = new LinkedTreeMap<>();
-        result.put("format_version", 9);
+        result.put("format_version", 10);
         result.put("pipelines", pipelineTransform.inverseTransform(pipeline));
         return YamlUtils.dump(result);
     }
 
     public JsonConfigCollection transform(Object rootObj, String location) {
         JsonConfigCollection partialConfig = new JsonConfigCollection();
         Map<String, Object> rootMap = (Map<String, Object>) rootObj;
+        // must obtain format_version first
+        int formatVersion = 1;
+        for (Map.Entry<String, Object> pe : rootMap.entrySet()) {
+            if ("format_version".equalsIgnoreCase(pe.getKey())) {
+                formatVersion = Integer.valueOf((String) pe.getValue());
+                partialConfig.updateFormatVersionFound(formatVersion);
+            }
+        }
         for (Map.Entry<String, Object> pe : rootMap.entrySet()) {
             if ("pipelines".equalsIgnoreCase(pe.getKey())) {
                 if (pe.getValue() == null)
                     continue;
                 Map<String, Object> pipelines = (Map<String, Object>) pe.getValue();
                 for (Map.Entry<String, Object> pipe : pipelines.entrySet()) {
                     try {
-                        JsonElement jsonPipeline = pipelineTransform.transform(pipe);
+                        JsonElement jsonPipeline = pipelineTransform.transform(pipe, formatVersion);
                         partialConfig.addPipeline(jsonPipeline, location);
                     } catch (Exception ex) {
                         partialConfig.addError(new PluginError(
@@ -65,9 +73,7 @@ public JsonConfigCollection transform(Object rootObj, String location) {
                                 String.format("Failed to parse environment %s; %s", env.getKey(), ex.getMessage()), location));
                     }
                 }
-            } else if ("format_version".equalsIgnoreCase(pe.getKey())) {
-                partialConfig.updateFormatVersionFound(Integer.valueOf((String) pe.getValue()));
-            } else if (!"common".equalsIgnoreCase(pe.getKey()))
+            } else if (!"common".equalsIgnoreCase(pe.getKey()) && !"format_version".equalsIgnoreCase(pe.getKey()))
                 throw new YamlConfigException(pe.getKey() + " is invalid, expected format_version, pipelines, environments, or common");
         }
         return partialConfig;