Specify custom API version on resources

[danielrbradley]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

In version 2 of the provider we're removing selected resources to reduce SDK size where the next version of the resource is compatible with the previous verion. In theory API behaviour can vary between versions and a user might want to use a specific version.

To ensure that the provider allows a user to interact with the full Azure surface area, we could add a string ApiVersion field to all resources which would allow the same SDK class to be used with other versions.

Initially this could be just a string field but could also be implemented to include an enum of known versions for that resource.

If the user specifies an invalid API version or a version that's incompatible with the resources shape of the SDK resource class then errors would just be propegated by the Azure API - we would not validate this. We could also consider disabling validation of properties if the version is specified in case transformations are also being used to manipulate the shape of the payload.

Affected area/feature

• Schema generation
• Provider

Use cases

• Using workload profiles with ManagedEnvironment throws error code "WorkloadProfileRelatedApiNotSupported" #2667
• Migration azure-native to v2 leads to ManagedCluster replacing #2669

[thomas11]

resolves #1877

[danielrbradley]

@thomas11 I don't think this does solve the specific use case in #1877 - that is for resources which don't exist in any way in the SDK rather than a different version of an existing resource.

[thomas11]

@danielrbradley the "resource doesn't exist in the SDK" case is covered by #68. Whereas #1877 is explicitly about API versions - although the semantics of the use case may differ. We should clarify this.

[danielrbradley]

@thomas11 ah yes your right - custom resource providers meaning API versions we don't support.

[danielrbradley]

Discussion points with @mikhailshilkov

1. Consider property naming to make it clear it's an "escape hatch". E.g. OverrideApiVersion
2. Add to the 2.0 migration guide for maintaining any v1 resource versions.
3. Add to the provider documentation to explain how it works & what it's for. Include details about using transformations to modify payload if required.

[cleverguy25]

Added to epic https://github.com/pulumi/home/issues/3552

[EronWright]

I would advocate for closing this ticket in favor of: #68

Is anyone able to provide a use case that would really be solved by adding an override api version?

[danielrbradley]

TODO: Re-spec this as a hidden escape hatch accessible via transforms.

[danielrbradley]

Updated design

One of the selling points of our Azure Native provider in v1 was to give complete access to the Azure API surface - including every API version. In v2 we had to remove some of these explicit versions due to the SDK size being too large. However, there is still need for users to be able to access all API versions even if the version is not specifically listed in the SDK. We are considering adding an output-only property for apiVersion (see #3766) to report back to users more clearly the specific API version the provided used to deploy the resource, but we don't want to surface this as a settable property because, for the majority of users, it would be a mistake to change this value.

In addition to the API version being customised, occasionally the schema for fields is incorrect or incomplete either due to a schema generation bug, an aspect of OpenAPI not being expressible in Pulumi Schema or the specifications being wrong or incomplete. In these scenarios it would be benificial for the user to be able to manually tweak the fields via use of the transforms resource option. However, currently at the point of preparing the SDK inputs to be sent as an Azure API request, we drop all fields which don't match the expected schema.

Example issues

• datafactor.Dataset does not support schema typeProperty. #2006

Therefore, we are proposing leveraging transforms for an 'escape hatch' for users needing to customise the request being sent to Azure's APIs.

Technical details

1. The apiVersion does not appear as an input in the SDK, but is settable via a transform.
2. When the apiVersion is specified, we will relax the SDK to API transformation and keep any extra properties or properties not matching the expected schema shape.

Example

Create a Dataset, but use a specific version and specify the typeProperties which is not in the SDK.

const vpc = new Dataset("vpc", {...}, {
    transforms: [args => {
        return {
            props: {
              ...args.props,
              apiVersion: "2018-06-01",
              typeProperties: {
                    schema: {
                        value: "@dataset().SchemaName",
                        type: "Expression"
                    },
                    table: {
                        value: "@dataset().TableName",
                        type: "Expression"
                    }
                }
            },
            opts: pulumi.mergeOptions(args.opts, { ignoreChanges: ["tags"] })
        }
    }],
});