[WIP] Ephemeral Resources prototype #35078

This is the new resource mode for ephemeral resources.

Ephemeral resources, declared using "ephemeral" blocks, represent objects that are instantiated only for the duration of a single Terraform phase, and are intended for uses such as temporary network tunnels or time-limited leases of sensitive values from stores such as HashiCorp Vault.

Similar to terraform_data, this is really just here to use as a placeholder when one needs an ephemeral resource for some reason but doesn't need any specific one. This might get removed before the ephemeral_values experiment gets stabilized. For now it's here to use as an initial testing vehicle since we don't have any mechanism for offering experimental features in the provider plugin protocol, whereas this provider is not a plugin.

This change is not shippable as-is because it changes the interpretation of any reference starting with "ephemeral.", which would previously have referred to a managed resource type belonging to a provider whose local name is "ephemeral". Therefore this initial attempt is only for prototyping purposes and would need to be modified in some way in order to be shippable. It will presumably need some sort of opt-in within the calling module so that the old interpretation can be preserved by default.

This is not yet sufficient to actually open/renew/close ephemeral resource instances, and so as of this commit a module including ephemeral resources will misbehave. Further work in subsequent commits.

There is not yet the needed support in the concrete evaluation data implementation, but this at least now knows to call it and collect the results.

We don't yet do anything useful when we get there, but we do at least fail in a vaguely-graceful way.

For now these graph nodes don't actually do anything, but the graph shape is at least plausible for what we'll need.

We now need to clean up any straggling ephemeral resource instances before we complete each graph walk, and ephemeral resource instances are ultimately owned by the graph walker, so the graph walker now has a Close method that's responsible for cleaning up anything that the walker owns which needs to be explicitly closed at the end of a walk.

Because ephemeralResourceCloseTransformer runs very late in the transform sequence, it's too late to get provider open and close nodes associated with it automatically. We don't actually need to worry about the provider _open_ dependency because our close node always depends on all of our open nodes and they will in turn depend on the provider open they need. But for close we need to delay closing the provider until all of the associated ephemeral resources have been closed, so we need to do a little fixup: If any of particular ephemeral resource's open nodes have provider close nodes depending on them, those provider close nodes should also depend on the ephemeral resource close node. That then describes that the provider should remain open for as long as at least one ephemeral resource instance owned by that provider remains live, which makes it okay for us to do our periodic background renew requests and our final close requests.

…alysis Previously we had a special interface graphNodeEphemeralResourceConsumer and a helper for implementing it in terms of GraphNodeReferencer, but for the moment we'll just use GraphNodeReferencer directly with that helper because that gives us broad coverage across many node types without having to make such sprawling changes just to support a prototype. The separated interface design might return later if we discover a need for a node to report that it uses an ephemeral resource without actually including any expression references for it, but we'll wait to see if that additional complexity is actually needed.

Ephemeral resources work quite differently than managed or data resources in that their instances live only in memory and are never persisted, and in that we need to handle the possibility of the object having become invalid by the time we're evaluating a reference expression. Since we're just prototyping ephemeral resources for now, this works as a totally separate codepath in the evaluator. The resource reference handling in the evaluator is long overdue for being reworked so that it doesn't depend so directly on the implementation details of how we keep track of resources, and the new ephemeral codepath is perhaps a simplified example of what that might look like in future, but for now it's used only for ephemeral resources to limit the invasiveness of this prototype.

I'm honestly not really sure yet how to explain _why_ ephemeral resource nodes are getting pruned when they shouldn't; for the sake of prototyping this is just a hard-coded special exception to just not consider them at all in the pruneUnusedNodesTransformer. The later ephemeralResourceCloseTransformer has its own logic for deciding that an ephemeral resource isn't actually needed in the current graph and pruning both their open and close nodes, so these will still get pruned but it will happen in different circumstances and based on a later form of the graph with more nodes and edges already present, thus preventing some cases of ephemeral resources being pruned when they shouldn't be.

The modules runtime should always use a different strategy to keep track of live ephemeral resource instances, and should never persist them in the plan or state. These checks are here just to reduce the risk that a bug in the modules runtime could inadvertently result in an ephemeral resource instance being persisted. This is a bit of a "defense-in-depth" strategy, because the state and plan types all have most of their fields exported and so we can't be sure that all modifications will go through the mutation methods.

This is just enough to skip writing and reading ephemeral resources and their instances in the plan and state, so that we can reach the code that manages them in their own separate data structure. This relies on the new idea of some resource modes not being persisted between rounds and not being persisted from plan to apply, although for now EphemeralResourceMode is the only mode that doesn't do both of those things.

Instead of a test for whether the type name is different than the one we expect, we'll use a switch statement. This does nothing for now, but a future commit will add a new ephemeral resource type that's intended only for prototyping, exploiting the fact that this particular provider can offer ephemeral resource types without us first extending the provider plugin protocol with that concept.

…esult

When a provider configuration is using an ephemeral resource, we need the closure of the resource instances to depend on the closure of the provider instance because otherwise we'll leave the ephemeral resource instance live only long enough to configure the provider, and that's useless for taking any other actions with the provider after it's been configured.

… type This is here only for the purposes of prototyping ephemeral resources. If we move forward with a "real" implementation then something like this would be better placed in a separate SSH provider, rather than built into Terraform CLI itself. This is just a basic implementation to get started with. It's probably not very robust and will probably need fixes and additions in future commits.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Ephemeral Resources prototype #35078

[WIP] Ephemeral Resources prototype #35078

Commits on Jun 19, 2024