Skip to content

Commit

Permalink
add context to projectmappings to allow for small modifications (e.g.…
Browse files Browse the repository at this point in the history
… name, namespace) without having to replace the entire template

Made resource_templates into a map, and made override merging of it (and context) be additive in nature so one can add a new mapping with a template without having to replace the existing ones
  • Loading branch information
wr0ngway committed Jun 17, 2021
1 parent 6ef09e1 commit 5d5646b
Show file tree
Hide file tree
Showing 14 changed files with 496 additions and 125 deletions.
45 changes: 10 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Parameterize the helm install with `--set appSettings.**` to control how kubetru
| projectMappings.root.key_selector | A regexp to limit the keys acted against (client-side). Supplies any named matches for template evaluation | string | "" | no |
| projectMappings.root.skip | Skips the generation of resources for the selected projects | flag | false | no |
| projectMappings.root.included_projects | Include the parameters from other projects into the selected ones. This can be recursive in a depth first fashion, so if A imports B and B imports C, then A will get B's and C's parameters. For key conflicts, if A includes B and B includes C, then the precendence is A overrides B overrides C. If A includes \[B, C], then the precendence is A overrides C overrides B. | list | [] | no |
| projectMappings.root.context | Additional variables made available to the resource templates. Can also be templates | string | [default](helm/kubetruth/values.yaml#L93-L129) | no |
| projectMappings.root.resource_templates | The templates to use in generating kubernetes resources (ConfigMap/Secrets/other) | string | [default](helm/kubetruth/values.yaml#L93-L129) | no |
| projectMappings.<override_name>.* | Define override mappings to override settings from the root selector for specific projects. When doing this on the command-line (e.g. for `helm install`), it may be more convenient to use `--values <file>` instead of `--set` for large data sets | map | {} | no |

Expand Down Expand Up @@ -136,16 +137,19 @@ them wakes it up from a polling sleep. This makes it quick and easy to test out
configuration changes without having a short polling interval.

To customize how the kubernetes resources are generated, edit the
`resource_templatse` property in the ProjectMappings. These templates are
`resource_templates` property in the ProjectMappings. These templates are
processed using the [Liquid template
language](https://shopify.github.io/liquid/), and can reference the following
liquid variables:

* `project` - The project name
* `project_heirarchy` - The `included_projects` tree this project includes (useful to debug when using complex `included_projects`)
* `debug` - Indicates if kubetruth is operating in debug (logging) mode.
* `parameters` - The CloudTruth parameters from the project
* `parameter_origins` - The projects each parameter originates from (useful to debug when using complex `included_projects`)
* `debug` - Indicates if kubetruth is operating in debug (logging) mode.
* `secrets` - The CloudTruth secrets from the project
* `secret_origins` - The projects each secret originates from (useful to debug when using complex `included_projects`)
* `context` - A hash of context variables supplied from ProjectMappings (useful to override portions of templates without having to replace them completely in an override)

In addition to the built in liquid filters, kubetruth also define a few custom
ones:
Expand Down Expand Up @@ -227,21 +231,9 @@ metadata:
spec:
scope: override
project_selector: funkyProject
resource_templates:
- |
apiVersion: v1
kind: ConfigMap
metadata:
namespace: notSoFunkyNamespace
name: notSoFunkyConfigMap
<snipped>
- |
apiVersion: v1
kind: Secret
metadata:
namespace: notSoFunkyNamespace
name: notSoFunkySecret
<snipped>
context:
resource_name: notSoFunkyConfigMap
resource_namespace: notSoFunkyNamespace
EOF
```

Expand Down Expand Up @@ -274,24 +266,7 @@ kubetruth-root root ^service 27m
$ kubectl describe pm kubetruth-root
Name: kubetruth-root
Namespace: default
Labels: ...
Annotations: ...
API Version: kubetruth.cloudtruth.com/v1
Kind: ProjectMapping
Metadata:
<snipped>
Spec:
resource_templates:
- |
<snipped>
- |
<snipped>
included_projects:
key_selector:
project_selector:
scope: root
skip: false
Events: <none>
<snipped>
```
## Development
Expand Down
9 changes: 7 additions & 2 deletions helm/helmv2/templates/projectmapping.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,14 @@ spec:
items:
type: string
description: Include the parameters from other projects into the selected ones. This can be recursive in a depth first fashion, so if A imports B and B imports C, then A will get B's and C's parameters. For key conflicts, if A includes B and B includes C, then the precendence is A overrides B overrides C. If A includes [B, C], then the precendence is A overrides C overrides B.
context:
type: object
additionalProperties:
type: string
description: Context variables that can be used by templates. The values can also be templates
resource_templates:
type: array
items:
type: object
additionalProperties:
type: string
description: The templates to use in generating kubernetes resources
additionalPrinterColumns:
Expand Down
9 changes: 7 additions & 2 deletions helm/kubetruth/crds/projectmapping.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,14 @@ spec:
items:
type: string
description: Include the parameters from other projects into the selected ones. This can be recursive in a depth first fashion, so if A imports B and B imports C, then A will get B's and C's parameters. For key conflicts, if A includes B and B includes C, then the precendence is A overrides B overrides C. If A includes [B, C], then the precendence is A overrides C overrides B.
context:
type: object
additionalProperties:
type: string
description: Context variables that can be used by templates. The values can also be templates
resource_templates:
type: array
items:
type: object
additionalProperties:
type: string
description: The templates to use in generating kubernetes resources
required:
Expand Down
13 changes: 9 additions & 4 deletions helm/kubetruth/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,17 @@ projectMappings:
key_selector: ""
skip: false
included_projects: []
context:
resource_name: "{{ project | dns_safe }}"
resource_namespace: ""
resource_templates:
- |
configmap: |
{%- if parameters.size > 0 %}
apiVersion: v1
kind: ConfigMap
metadata:
name: "{{ project | dns_safe }}"
name: "{{ context.resource_name }}"
namespace: "{{ context.resource_namespace }}"
labels:
version: "{{ parameters | sort | to_json | sha256 | slice: 0, 7 }}"
annotations:
Expand All @@ -109,12 +113,13 @@ projectMappings:
{%- endfor %}
{%- endif %}
- |
secret: |
{%- if secrets.size > 0 %}
apiVersion: v1
kind: Secret
metadata:
name: "{{ project | dns_safe }}"
name: "{{ context.resource_name }}"
namespace: "{{ context.resource_namespace }}"
labels:
version: "{{ secrets | sort | to_json | sha256 | slice: 0, 7 }}"
annotations:
Expand Down
21 changes: 14 additions & 7 deletions lib/kubetruth/config.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ class DuplicateSelection < Kubetruth::Error; end
:key_selector,
:skip,
:included_projects,
:context,
:resource_templates,
keyword_init: true
) do
Expand All @@ -23,17 +24,17 @@ def initialize(*args, **kwargs)

def convert_types(hash)
selector_key_pattern = /_selector$/
template_key_pattern = /_templates?$/
template_key_pattern = /_template$/
hash.merge(hash) do |k, v|
case k
when selector_key_pattern
Regexp.new(v)
when template_key_pattern
if k.ends_with?('s')
v.collect {|t| Kubetruth::Template.new(t) }
else
Kubetruth::Template.new(v)
end
Kubetruth::Template.new(v)
when /^resource_templates$/
Hash[v.collect {|k, t| [k.to_s, Kubetruth::Template.new(t)] }]
when /^context$/
Kubetruth::Template::TemplateHashDrop.new(v)
else
v
end
Expand All @@ -48,6 +49,7 @@ def convert_types(hash)
key_selector: '',
skip: false,
included_projects: [],
context: {},
resource_templates: []
}.freeze

Expand All @@ -66,7 +68,12 @@ def load

config = DEFAULT_SPEC.merge(root_mapping)
@root_spec = ProjectSpec.new(**config)
@override_specs = overrides.collect { |o| ProjectSpec.new(**config.merge(o)) }
logger.debug { "ProjectSpec for root mapping: #{@root_spec.inspect}"}
@override_specs = overrides.collect do |o|
spec = ProjectSpec.new(**config.deep_merge(o))
logger.debug { "ProjectSpec for override mapping: #{spec.inspect}"}
spec
end
config
end
end
Expand Down
15 changes: 11 additions & 4 deletions lib/kubetruth/etl.rb
Original file line number Diff line number Diff line change
Expand Up @@ -120,16 +120,18 @@ def apply
config_origins = Hash[param_origins_parts[true] || []]
secret_origins = Hash[param_origins_parts[false] || []]

project.spec.resource_templates.each_with_index do |template, i|
logger.debug { "Processing template #{i}/#{project.spec.resource_templates.size}" }
project.spec.resource_templates.each_with_index do |pair, i|
template_name, template = *pair
logger.debug { "Processing template '#{template_name}' (#{i}/#{project.spec.resource_templates.size})" }
resource_yml = template.render(
project: project.name,
project_heirarchy: project.heirarchy,
debug: logger.debug?,
parameters: config_param_hash,
parameter_origins: config_origins,
secrets: secret_param_hash,
secret_origins: secret_origins
secret_origins: secret_origins,
context: project.spec.context
)
parsed_yml = YAML.safe_load(resource_yml)
if parsed_yml
Expand All @@ -148,8 +150,13 @@ def params_to_hash(param_list)

def kube_apply(parsed_yml)
kind = parsed_yml["kind"]
namespace = parsed_yml["metadata"]["namespace"] || kubeapi.namespace
name = parsed_yml["metadata"]["name"]
namespace = parsed_yml["metadata"]["namespace"]
if namespace.blank?
namespace = parsed_yml["metadata"]["namespace"] = kubeapi.namespace
end

kubeapi.set_managed(parsed_yml)

ident = "'#{namespace}:#{kind}:#{name}'"
logger.info("Applying kubernetes resource #{ident}")
Expand Down
13 changes: 5 additions & 8 deletions lib/kubetruth/kubeapi.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ def initialize(namespace: nil, token: nil, api_url: nil)
token_path = '/var/run/secrets/kubernetes.io/serviceaccount/token'

@namespace = namespace.present? ? namespace : File.read(namespace_path).chomp
@labels = {MANAGED_LABEL_KEY => MANAGED_LABEL_VALUE}

@auth_options = {}
if token
Expand Down Expand Up @@ -64,14 +63,14 @@ def ensure_namespace(ns = namespace)
end

def under_management?(resource)
labels = resource&.metadata&.labels
labels.nil? ? false : resource.metadata.labels[MANAGED_LABEL_KEY] == MANAGED_LABEL_VALUE
labels = resource&.[]("metadata")&.[]("labels")
labels.nil? ? false : resource["metadata"]["labels"][MANAGED_LABEL_KEY] == MANAGED_LABEL_VALUE
end

def set_managed(resource)
resource.metadata ||= {}
resource.metadata.labels ||= {}
resource.metadata.labels[MANAGED_LABEL_KEY] = MANAGED_LABEL_VALUE
resource["metadata"] ||= {}
resource["metadata"]["labels"] ||= {}
resource["metadata"]["labels"][MANAGED_LABEL_KEY] = MANAGED_LABEL_VALUE
end

def get_resource(resource_name, name, namespace=nil)
Expand All @@ -80,9 +79,7 @@ def get_resource(resource_name, name, namespace=nil)

def apply_resource(resource)
resource = Kubeclient::Resource.new(resource) if resource.is_a? Hash
set_managed(resource)
resource_name = resource.kind.downcase.pluralize
resource.metadata.namespace ||= self.namespace
client.apply_entity(resource_name, resource, field_manager: "kubetruth")
end

Expand Down
38 changes: 36 additions & 2 deletions lib/kubetruth/template.rb
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,30 @@ class Template
class Error < ::StandardError
end

class TemplateHashDrop < Liquid::Drop

attr_reader :source

def initialize(template_hash)
@source = template_hash.stringify_keys
@parsed = {}
end

def liquid_method_missing(key)
if @source.has_key?(key)
@parsed[key] ||= Template.new(@source[key])
@parsed[key].render(@context)
else
super
end
end

def inspect
{source: @source, parsed: @parsed}.inspect
end

end

module CustomLiquidFilters

# From kubernetes error message
Expand Down Expand Up @@ -93,7 +117,7 @@ def initialize(template_source)

INDENT = (" " * 2)

def render(**kwargs)
def render(*args, **kwargs)
begin

logger.debug do
Expand All @@ -104,7 +128,7 @@ def render(**kwargs)
msg
end

result = @liquid.render!(kwargs.stringify_keys, strict_variables: true, strict_filters: true)
result = @liquid.render!(*args, kwargs.stringify_keys, strict_variables: true, strict_filters: true)

logger.debug do
msg = "Rendered template:\n"
Expand All @@ -115,6 +139,16 @@ def render(**kwargs)
result

rescue Liquid::Error => e
indent = " "
msg = "Template failed to render:\n"
@source.lines.each {|l| msg << (indent * 2) << l }
msg << indent << "with error message:\n" << (indent * 2) << "#{e.message}"
if e.is_a?(Liquid::UndefinedVariable)
msg << "\n" << indent << "and variable context:\n"
msg << (indent * 2) << kwargs.inspect
end
raise Error, msg

raise Error, "Template failed to render: #{e.message}"
end
end
Expand Down
Loading

0 comments on commit 5d5646b

Please sign in to comment.