[WIP] Ephemeral Resources prototype #35078
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apparentlymart
force-pushed
the
f-ephemeral-values
branch
from
f5841fc
to
2ce8606
Compare
apparentlymart
force-pushed
the
f-ephemeral-values
branch
from
2ce8606
to
5c7deeb
Compare
apparentlymart
force-pushed
the
f-ephemeral-values
branch
from
be64383
to
aaa345f
Compare
apparentlymart
force-pushed
the
f-ephemeral-values
branch
from
25072c8
to
2ab8071
Compare
apparentlymart
force-pushed
the
f-ephemeral-values
branch
from
95e1621
to
59ea75b
Compare
apparentlymart
force-pushed
the
f-ephemeral-values
branch
from
059c2f9
to
e6c8430
Compare
This is the new resource mode for ephemeral resources.
Ephemeral resources, declared using "ephemeral" blocks, represent objects that are instantiated only for the duration of a single Terraform phase, and are intended for uses such as temporary network tunnels or time-limited leases of sensitive values from stores such as HashiCorp Vault.
Similar to terraform_data, this is really just here to use as a placeholder when one needs an ephemeral resource for some reason but doesn't need any specific one. This might get removed before the ephemeral_values experiment gets stabilized. For now it's here to use as an initial testing vehicle since we don't have any mechanism for offering experimental features in the provider plugin protocol, whereas this provider is not a plugin.
This change is not shippable as-is because it changes the interpretation of any reference starting with "ephemeral.", which would previously have referred to a managed resource type belonging to a provider whose local name is "ephemeral". Therefore this initial attempt is only for prototyping purposes and would need to be modified in some way in order to be shippable. It will presumably need some sort of opt-in within the calling module so that the old interpretation can be preserved by default.
This is not yet sufficient to actually open/renew/close ephemeral resource instances, and so as of this commit a module including ephemeral resources will misbehave. Further work in subsequent commits.
There is not yet the needed support in the concrete evaluation data implementation, but this at least now knows to call it and collect the results.
We don't yet do anything useful when we get there, but we do at least fail in a vaguely-graceful way.
For now these graph nodes don't actually do anything, but the graph shape is at least plausible for what we'll need.
We now need to clean up any straggling ephemeral resource instances before we complete each graph walk, and ephemeral resource instances are ultimately owned by the graph walker, so the graph walker now has a Close method that's responsible for cleaning up anything that the walker owns which needs to be explicitly closed at the end of a walk.
Because ephemeralResourceCloseTransformer runs very late in the transform sequence, it's too late to get provider open and close nodes associated with it automatically. We don't actually need to worry about the provider _open_ dependency because our close node always depends on all of our open nodes and they will in turn depend on the provider open they need. But for close we need to delay closing the provider until all of the associated ephemeral resources have been closed, so we need to do a little fixup: If any of particular ephemeral resource's open nodes have provider close nodes depending on them, those provider close nodes should also depend on the ephemeral resource close node. That then describes that the provider should remain open for as long as at least one ephemeral resource instance owned by that provider remains live, which makes it okay for us to do our periodic background renew requests and our final close requests.
…alysis Previously we had a special interface graphNodeEphemeralResourceConsumer and a helper for implementing it in terms of GraphNodeReferencer, but for the moment we'll just use GraphNodeReferencer directly with that helper because that gives us broad coverage across many node types without having to make such sprawling changes just to support a prototype. The separated interface design might return later if we discover a need for a node to report that it uses an ephemeral resource without actually including any expression references for it, but we'll wait to see if that additional complexity is actually needed.
Ephemeral resources work quite differently than managed or data resources in that their instances live only in memory and are never persisted, and in that we need to handle the possibility of the object having become invalid by the time we're evaluating a reference expression. Since we're just prototyping ephemeral resources for now, this works as a totally separate codepath in the evaluator. The resource reference handling in the evaluator is long overdue for being reworked so that it doesn't depend so directly on the implementation details of how we keep track of resources, and the new ephemeral codepath is perhaps a simplified example of what that might look like in future, but for now it's used only for ephemeral resources to limit the invasiveness of this prototype.
I'm honestly not really sure yet how to explain _why_ ephemeral resource nodes are getting pruned when they shouldn't; for the sake of prototyping this is just a hard-coded special exception to just not consider them at all in the pruneUnusedNodesTransformer. The later ephemeralResourceCloseTransformer has its own logic for deciding that an ephemeral resource isn't actually needed in the current graph and pruning both their open and close nodes, so these will still get pruned but it will happen in different circumstances and based on a later form of the graph with more nodes and edges already present, thus preventing some cases of ephemeral resources being pruned when they shouldn't be.
The modules runtime should always use a different strategy to keep track of live ephemeral resource instances, and should never persist them in the plan or state. These checks are here just to reduce the risk that a bug in the modules runtime could inadvertently result in an ephemeral resource instance being persisted. This is a bit of a "defense-in-depth" strategy, because the state and plan types all have most of their fields exported and so we can't be sure that all modifications will go through the mutation methods.
This is just enough to skip writing and reading ephemeral resources and their instances in the plan and state, so that we can reach the code that manages them in their own separate data structure. This relies on the new idea of some resource modes not being persisted between rounds and not being persisted from plan to apply, although for now EphemeralResourceMode is the only mode that doesn't do both of those things.
Instead of a test for whether the type name is different than the one we expect, we'll use a switch statement. This does nothing for now, but a future commit will add a new ephemeral resource type that's intended only for prototyping, exploiting the fact that this particular provider can offer ephemeral resource types without us first extending the provider plugin protocol with that concept.
When a provider configuration is using an ephemeral resource, we need the closure of the resource instances to depend on the closure of the provider instance because otherwise we'll leave the ephemeral resource instance live only long enough to configure the provider, and that's useless for taking any other actions with the provider after it's been configured.
… type This is here only for the purposes of prototyping ephemeral resources. If we move forward with a "real" implementation then something like this would be better placed in a separate SSH provider, rather than built into Terraform CLI itself. This is just a basic implementation to get started with. It's probably not very robust and will probably need fixes and additions in future commits.
apparentlymart
force-pushed
the
f-ephemeral-values
branch
from
9b8cb54
to
41fa1e1
Compare
apparentlymart
changed the title
|
Okay... I did have some spare time today after all and so I cherry-picked the relevant subset of the changes one at a time and it wasn't so bad once I had more awareness of what each conflict was related to. 😅 Therefore this PR is now a draft primarily of Ephemeral Resources in particular, rather than ephemeral values in general. For ephemeral input variables and output values, refer to #35273 instead. There is not yet any prototype of write-only attributes at the time I'm writing this. |
|
Others on the Terraform team are going to design and implement the final version of these ideas, and so this PR will not proceed any further but is hopefully still useful as a reference. |
|
I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is another attempt at introducing to Terraform the idea of objects and values being "ephemeral", which means something like "lives only for the duration of one Terraform phase".
Terraform already has at least two concepts that meet this definition, despite us not previously naming it:
providerblocks): Terraform re-evaluates the arguments in aproviderblock separately during the plan and apply phases, and doesn't mind if the configuration is different between the two as long as the apply-time configuration allows performing the actions that were proposed during the plan phase.provisionerandconnectionblocks): Terraform fully evaluates these only during the apply phase, so they aren't really considered during the plan phase at all, aside from basic static validation.However, because the idea of "ephemeral" is not available in the rest of the language, it's tough to actually benefit from this ephemerality. This prototype aims to introduce "ephemeral" as a cross-cutting concern supported broadly across the language.
Ephemeral Values
The most fundamental idea is that values used in expressions can either be ephemeral or non-ephemeral. This is an idea similar to "sensitive" in that Terraform will perform dynamic analysis such that any value derived from an ephemeral value is itself ephemeral. Ephemeral values can then be used only in parts of the language which would not require persisting the value either between the plan phase and the apply phase, or from one plan/apply round to the next.
Considering only pre-existing language features, ephemeral values can be freely used in
providerblocks,provisionerblocks,connectionblocks, and in local values. The following sections describe some new additions that either accept or produce ephemeral values.resourceblocks (aside from special nested parts like the aforementionedprovisionerblocks) do not accept ephemeral values, because preserving resource configuration unchanged between the plan and apply phases is a fundamental part of how Terraform works to keep its promise of either doing what the plan described or returning an error explaining why that's not possible.Because ephemeral values are not expected to persist from plan to apply or between plan/apply rounds, there is no need to save them in saved plan files or state snapshots, thus finally giving a plausible answer for what to do about #516, which has been on my mind since long before I worked at HashiCorp.
Ephemeral Input Variables
An ephemeral input variable is, in the most general terms, just an input variable that is declared as accepting ephemeral values. A non-ephemeral input variable cannot accept ephemeral values, while an ephemeral value will accept both ephemeral and non-ephemeral values but the value will always be treated as ephemeral when used inside the declaring module.
The main interesting case is when a root module declares an ephemeral input variable. In that case, Terraform will no longer remember the value for the variable provided during planning and will instead expect any ephemeral variable set during the plan step to be provided again -- possibly with a different value -- during the apply step.
The primary goal of this is to be able to use input variables to set arguments in ephemeral contexts. For example, an input variable that's both ephemeral and sensitive could provide a JSON Web Token to be used when configuring a specific provider, and then automation around Terraform could provide separate JSON Web Tokens across the plan and apply phases so that the apply phase isn't subject to the expiration time for the plan-time JWT, and so that the plan-time JWT doesn't get persisted to disk as part of a saved plan.
Ephemeral Output Values
An ephemeral output value is essentially the opposite of an ephemeral input variable, allowing a module to expose an ephemeral value to its caller. As with input variables, a non-ephemeral output value will reject having an ephemeral value assigned to it. An ephemeral output value can have both ephemeral and non-ephemeral values assigned to it, but the calling module will always see it as ephemeral.
To start the utility of this is limited just to echoing back values derived from ephemeral input variables, since nothing else I've described so far actually produces ephemeral values. However, allowing this is important to ensure that ephemeral values are supported symmetrically and will cooperate well with all other language features.
Ephemeral Resources
The final idea in this prototype -- one which this prototype probably won't explore fully just yet, and introduce only just enough to validate that it fits in well with everything else -- is a new resource mode for representing remote objects that are ephemeral themselves.
Terraform currently has two "resource modes": managed resources (
resourceblocks) describe objects that Terraform is directly managing, while data resources (datablocks) describe objects that are managed elsewhere that the current configuration depends on. But in both cases the assumption is that those objects persist in some sense from plan to apply and from one plan/apply round to the next, and that Terraform is supposed to detect and react to any changes to those objects and therefore needs to persist information about them itself.Ephemeral resources, (
ephemeralblocks) on the other hand, represent objects that -- at least, as far as Terraform is concerned -- exist only briefly during a single Terraform phase, and then get cleaned up once the phase is complete. This idea is an evolution of some much earlier design work I did before I even worked at HashiCorp 😀 in relation to #8367, which was about establishing temporary SSH tunnels, and the HashiCorp Vault provider I wrote in #9158 (which evolved into today's officialhashicorp/vault).The general idea of ephemeral resources, then, is that their lifecycle includes three events:
OpenEphemeral: Prepares the object for use. For some kinds of objects this would represent a "create" action, but for others it might just open a temporary session to something that already exists, such as in the SSH tunnel use-case.This operation is the one that establishes the result attributes that can be accessed from other parts of the module where the resource is declared. All of these results would be ephemeral values, so that they can vary from plan to apply. For example, opening an SSH tunnel is likely to cause a different local TCP port number to be allocated each time, and so consistency between plan and apply phases is not expected.
RenewEphemeral: Some ephemeral remote objects need to be periodically refreshed in order to stay "live", such as leases for Vault secrets.This optional operation is therefore opted into by the provider's
OpenEphemeralresponse, by providing a private set of data that should be sent back to the provider'sRenewEphemeralimplementation and a deadline before which Terraform must renew it. The provider can then do whatever is needed to keep the object from expiring, and optionally return another renew request with a new deadline in order to repeat this renewal process.CloseEphemeral: Once Terraform has completed work for all objects that refer to the ephemeral resource, this operation gives the provider an explicit signal that the object is not longer required so that it can be promptly destroyed or invalidated.This detail is particularly helpful for the Vault provider and fixes a limitation I ran into immediately back in 2016: a dynamic secret fetched using a
datablock can never have its lease explicitly terminated, because data resources were intended only to read information about an object someone else is managing, not to directly manage an object (a Vault lease).Because the results from ephemeral resources are ephemeral values, they're primarily useful in configuration for other ephemeral objects:
providerblocks,provisioner/connectionblocks, and of course otherephemeralblocks.Actually changing the provider protocol and implementing real providers is not in scope for my initial prototyping work here, and so I intend to prototype this in a more limited way that just emulates how this mechanism might behave, so we can see how well it interacts with the rest of the language and the other ephemeral values discussed here.
I've also been considering a mechanism to allow managed resource types to declare individual arguments as being "write-only", such as for an RDS database password that only needs to be provided during creation and should not be provided again unless the operator actually intends to reset it. I don't intend to prototype that in here, but I intend to lay the foundations for it by having a convention that ephemeral input values and write-only arguments both treat
nullas meaning "don't set or change" and non-null as "set or change", thereby creating a small imperative-shaped niche in the otherwise-declarative Terraform Language to allow for using Terraform to manage objects that have write-only (typically, sensitive) arguments without needing to persist them in plan and state.I'm still working on this, so not everything described above is in here yet, but the foundations for ephemeral values themselves are already in. I've opened this draft largely just because I need to put this work down for a while for a team offsite and don't want to lose the context.