[WIP] Generic provider template #1140

blampe · 2024-11-12T00:14:36Z

This is a very WIP fork of our bridged workflows which is intended to be used by native, component, or other miscellaneous providers not currently managed by ci-mgmt.

There are two primary design principles which deviate from the bridged template:

make targets define the entire contract between a provider and ci-mgmt. We do not assume anything about the provider, e.g. it could be implemented in a language other than Go.
The provider is responsible for defining its own overrides and customizations, and ci-mgmt is responsible for exposing the appropriate hooks (Allow make target customization #1131).

For example, if the provider needs some additional tooling then it is responsible for ensuring the tool is installed. Historically these sorts of customizations have been handled as one-offs in ci-mgmt, but we want to push this down to the make target or into the test itself. Not only does this make ci-mgmt easier to maintain and refactor, but it also forces us to write tests that are more easily reproducible in local environments. The SSH key handling between the docker and docker-build providers is illustrative.

I've been testing this in EKS and haven't gotten tests passing reliably yet. I was planning on consolidating the test into a shared action once I got that far, but also feel free to do that sooner than later!

EKS was already sharding tests so I've opted to keep that behavior here -- a provider just needs to implement a test_shard target which takes a TESTS and TAG env var. I've tweaked it somewhat so the sharding is totally arbitrary, which in practice just means we need to install all the SDKs instead of one at a time.

Other differences:

We don't install gotestfmt. (Ref Remove gotestfmt (second PR) #1072)
We don't have setupScript, extraTests, preTest, etc.

TODO:

Make e2e test path configurable (currently hard-coded to examples).
Make shard determination configurable and local (currently a workflow step).
Consolidate test action.

danielrbradley

Direction & progress looks good. It would probably be worth writing down a design for the required makefile targets and artifacts while doing this work - perhaps in a README somewhere in this repo - which can serve as a design doc too.

danielrbradley · 2024-11-12T10:15:12Z

provider-ci/internal/pkg/templates/generic/.github/workflows/build_provider.yml

+      - name: Download schema-embed.json
+        uses: #{{ .Config.ActionVersions.DownloadArtifact }}#
+        with:
+          # Use a pattern to avoid failing if the artifact doesn't exist
+          pattern: schema-embed.*
+          # Avoid creating directories for each artifact
+          merge-multiple: true
+          path: provider/cmd/pulumi-resource-#{{ .Config.Provider }}#/schema-embed.json


Not all providers use schema-embed right now. It's only realy a performance optimisation which could be implemented internally within the provider rather than needing CI to have insight of.

We mostly still have the original schema commited (though this has been taken out Azure Native due to the file being too big for GitHub, and LFS causing a load of other problems) but I think it would be more correct to treat the schema like a build artifact as you're doing here, though it would be nicer to not assume the Go provider directory structure if possible within the CI job. I think this could be abstracted by the makefile if needed.

Prerequisites writes all artifacts to the bin directory including the embeddable schema if it needs it.

Restore the bin directory in this subsequent job.

Within the provider_dist target, copy the embeddable schema into the provider directory before building.

danielrbradley · 2024-11-12T10:17:49Z

provider-ci/internal/pkg/templates/generic/.github/workflows/build_provider.yml

+          merge-multiple: true
+          path: provider/cmd/pulumi-resource-#{{ .Config.Provider }}#/schema-embed.json
+      - name: Build & package provider
+        run: make provider_dist-${{ matrix.platform.os }}-${{ matrix.platform.arch }}


Perhaps we should use the file target here to match the path we're going to upload rather than having the assumption of the path this phony target will write to:

make dist/pulumi-resource-#{{ .Config.Provider }}#-v${{ inputs.version }}-${{ matrix.platform.os }}-${{ matrix.platform.arch }}.tar.gz

danielrbradley · 2024-11-12T10:30:02Z

provider-ci/internal/pkg/templates/generic/.github/workflows/build_sdk.yml

+      - name: Install plugins
+        run: make install_plugins


This is a bit of an odd step. I know for TF providers it's related to installing other bridged TF providers which the bridge will use for examples generation, but for a generic template we might want something a little more generic such as make prepare or something that allways runs at the start of the job before restoring any targets or doing make --touch ... etc.

Yeah, I was also wondering about this because the CLI should automatically download these plugins anyway. Related - these plugins eat a ton of space! pulumi/pulumi#17718

I think for bridged providers, if they're opted into using the converter plugin, then it explicitly disables auto-download because we need to have the provider version pinned, so failing fast is good. For generic providers, perhaps we can just let the provider figure out what it needs and how it will install them internally within the makefile.

danielrbradley · 2024-11-12T11:34:39Z

provider-ci/internal/pkg/templates/generic/.github/workflows/run-acceptance-tests.yml

+        # otherwise use the current SHA for any other type of build.
+        sha: ${{ github.event.pull_request.head.sha || github.sha }}
+
+  # TODO: Extract into shared action.


danielrbradley · 2024-11-12T11:36:58Z

provider-ci/internal/pkg/templates/generic/.github/workflows/license.yml

@@ -0,0 +1,48 @@
+# WARNING: This file is autogenerated - changes will be overwritten if not made via https://github.com/pulumi/ci-mgmt
+
+name: license_check


I think this should already be included from the provider template .. unless there's something subtle we're tweaking here I didn't spot?

Ah yeah, I duplicated it here because of #1133

danielrbradley · 2024-11-12T11:39:50Z

provider-ci/internal/pkg/generate.go

+	case "generic":
+		return []string{"provider", "pulumi-provider", "generic"}, nil


When we use this template for the native boilerplate, we'll probably add external-generic which will exclude the pulumi-provider sub-template.

Or maybe "third-party"?

Just above is the external-bridged-provider template which is like bridged-provider but with bits removed for third parties. Whatever name we use should aim to keep these consistent.

danielrbradley

Love the design in the README. That's super helpful 💜

Sharding is interesting, but doesn't feel it's quite there yet.

Relying on phony targets vs files is worth a quick think, but the phony targets work fine if in doubt.

danielrbradley · 2024-11-21T15:20:36Z

provider-ci/internal/pkg/templates/README.md

+* `make install_plugins`: (TODO: Use a more generic `make prepare` or just drop
+  this.)
+* `make schema`: Ensures generated schema is in place.
+* `make provider`: Builds the provider binary.


Suggested change

* `make provider`: Builds the provider binary.

* `make provider`: Builds the provider binary for the purpose of local testing. The final distributable binary is built using `make provider_dist-*`.

This could also be a file target, though slightly more tricky because the name of the binary changes per provider, so perhaps just documenting it's in the bin directory is enough.

danielrbradley · 2024-11-21T15:23:14Z

provider-ci/internal/pkg/templates/README.md

+
+* `make install_plugins`: (TODO: Use a more generic `make prepare` or just drop
+  this.)
+* `make schema`: Ensures generated schema is in place.


This could be a file target of bin/schema.json which can then be expected to be in place and included as a publishable artifact along with the provider binaries for reference (and perhaps integration with the registry in the future). This could still also output the schema to any other required locations as needed.

danielrbradley · 2024-11-21T17:47:27Z

provider-ci/internal/pkg/templates/README.md

+* `make shard`: This target takes two environment variables -- `$SHARD_TOTAL`
+  and `$SHARD_INDEX` -- and is responsible for determining tests to run for
+  this shard. It will probably mutate the environment in some way, for example
+  by appending to `$GITHUB_ENV`, in order to inform the `test_shard` target.
+* `make test_shard`: This target is responsible for executing tests identified
+  in the `shard` target.


I'm a little confused why we have two targets here. It feels like we shouldn't need make shard at all .. and we don't want to have to tell the provider author to know how to interact with github env.

Perhaps we should be just using make test or make integration_test with variables for SHARD_TOTAL and SHARD_INDEX e.g. make integration_test SHARD_TOTAL=5 SHARD_INDEX=3.

It would also be nice to link from here to an example of how to do sharding with go test.

danielrbradley · 2024-12-03T15:53:25Z

Done a bit more experimenting around the test sharding and am pretty sure we can do this with a single target with optional make variables.

Here's how we can adapt our current bridge makefile to support sharding (or runing specific tests locally):

TEST_RUN_ARG =
ifneq "$(origin TEST_RUN)" "undefined"
  TEST_RUN_ARG = -run "${TEST_RUN}"
else ifneq ("$(origin TEST_SHARD_TOTAL)|$(origin TEST_SHARD_INDEX)", "undefined|undefined")
	TEST_RUN_ARG = -run "$$(go run github.com/blampe/shard@9d1f3b21786e18caa1989e19502595143985d61b --total "$(TEST_SHARD_TOTAL)" --index "$(TEST_SHARD_INDEX)" --output env)"
endif

test: export PATH := $(WORKING_DIR)/bin:$(PATH)
test:
	cd examples && go test -v -tags=all -parallel $(TESTPARALLELISM) -timeout 2h $(TEST_RUN_ARG)
.PHONY: test

TEST_PROVIDER_RUN_ARG =
ifneq "$(origin TEST_RUN)" "undefined"
  TEST_PROVIDER_RUN_ARG = -run "${TEST_RUN}"
else ifneq ("$(origin TEST_SHARD_TOTAL)|$(origin TEST_SHARD_INDEX)", "undefined|undefined")
	TEST_PROVIDER_RUN_ARG = -run "$$(go run github.com/blampe/shard@9d1f3b21786e18caa1989e19502595143985d61b --total "$(TEST_SHARD_TOTAL)" --index "$(TEST_SHARD_INDEX)" --output env)"
endif

Using -n to just print commands to demonstrate usage:

make -n test TEST_RUN="Things"
cd examples && go test -v -tags=all -parallel 10 -timeout 2h -run "Things"

make -n test_provider TEST_RUN="Things"
cd provider && go test -v -short ./... -parallel 10 -run "Things"

make -n test TEST_SHARD_INDEX=2 TEST_SHARD_TOTAL=5
cd examples && go test -v -tags=all -parallel 10 -timeout 2h -run "$(go run github.com/blampe/shard@9d1f3b21786e18caa1989e19502595143985d61b --total "5" --index "2" --output env)"

make -n test_provider TEST_SHARD_INDEX=2 TEST_SHARD_TOTAL=5
cd provider && go test -v -short ./... -parallel 10 -run "$(go run github.com/blampe/shard@9d1f3b21786e18caa1989e19502595143985d61b --total "5" --index "2" --output env)"

danielrbradley · 2024-12-06T11:42:22Z

Next thoughts on the sharding ... I think we should roll it out as a smallish change to bridged providers first to solidify the design.

In order to get this live, we should create a version of the shard program in the pulumi organisation.
How are we planning to upgrade the version of the shard tool that we depend on? Can we hook this into Renovate?
In the implementation of the shard tool - is there any way to leverage Go's own test discovery process so we're sure the tool behaves consistently with running go test?
Related to (3): I think there's potentially an issue here around when there's modules within modules (e.g. with the shim or upstream directories).

This onboards the provider to new workflows consistent with pulumi/ci-mgmt#1140. This doesn't change anything with regard to the release process, but it does bring the provider up-to-speed with current best practices and it will make it easier to automatically manage with `ci-mgmt` in the future. Notable changes: * Java publishing * Embedding versioning (`respectSchemaVersion`) * Sharded tests * Goreleaser replaced by `pulumi/pulumi-package-publisher`

blampe marked this pull request as draft November 12, 2024 00:14

blampe requested a review from danielrbradley November 12, 2024 00:14

blampe changed the base branch from master to blampe/typed-config November 12, 2024 00:15

danielrbradley reviewed Nov 12, 2024

View reviewed changes

Base automatically changed from blampe/typed-config to master November 15, 2024 21:41

mjeffryes assigned blampe Nov 15, 2024

blampe added 15 commits November 19, 2024 10:16

Move COC to pulumi-provider

c573976

Add a generic template derived from bridged-provider

853581e

Generate test-providers/eks

71823a2

Remove some bridge workflows

67ce814

fix license

f20013d

no tfgen binary

db9d82f

use codecov instead of s3

ca29c24

Remove extraTests, sshPrivateKey, preTest, gotestfmt

6d51cba

remove some upstream/tf references

26d37d0

don't shard by language

96a9aa2

remove default makefile targets for now

5322514

regenerate eks

3a9de3b

consolidate test workflow

b0c4b1c

include env vars in tests

0dc27e2

readme

ce7311e

blampe force-pushed the blampe/eks branch from c329cbf to ce7311e Compare November 19, 2024 18:17

blampe requested a review from danielrbradley November 19, 2024 18:18

danielrbradley reviewed Nov 21, 2024

View reviewed changes

blampe mentioned this pull request Dec 5, 2024

ci-mgmt onboarding pulumi/pulumi-pulumiservice#446

Merged

danielrbradley mentioned this pull request Dec 6, 2024

Turn on test sharding for k8s and docker-build #1058

Open

DON'T FORGET TO ONLY HAVE THE IF ON PR WORKFLOW

e74c987

This was referenced Dec 12, 2024

Allow bridged providers to opt-in to sharding #1217

Open

[Epic] Standardize on provider-ci for all providers #1101

Open

Merge branch 'master' of github.com:pulumi/ci-mgmt into blampe/eks

81ee58a

blampe mentioned this pull request Jan 4, 2025

[WIP] Alternative approach for native providers #1268

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Generic provider template #1140

[WIP] Generic provider template #1140

blampe commented Nov 12, 2024 •

edited

Loading

danielrbradley left a comment

danielrbradley Nov 12, 2024

danielrbradley Nov 12, 2024

danielrbradley Nov 12, 2024

blampe Nov 12, 2024

danielrbradley Nov 12, 2024

danielrbradley Nov 12, 2024

danielrbradley Nov 12, 2024

blampe Nov 12, 2024

danielrbradley Nov 12, 2024

blampe Nov 12, 2024

danielrbradley Nov 12, 2024

danielrbradley left a comment

danielrbradley Nov 21, 2024

danielrbradley Nov 21, 2024

danielrbradley Nov 21, 2024

danielrbradley Nov 21, 2024

danielrbradley commented Dec 3, 2024 •

edited

Loading

danielrbradley commented Dec 6, 2024

		@@ -0,0 +1,48 @@
		# WARNING: This file is autogenerated - changes will be overwritten if not made via https://github.com/pulumi/ci-mgmt

		name: license_check

		case "generic":
		return []string{"provider", "pulumi-provider", "generic"}, nil

	* `make provider`: Builds the provider binary.
	* `make provider`: Builds the provider binary for the purpose of local testing. The final distributable binary is built using `make provider_dist-*`.

[WIP] Generic provider template #1140

Are you sure you want to change the base?

[WIP] Generic provider template #1140

Conversation

blampe commented Nov 12, 2024 • edited Loading

danielrbradley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielrbradley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielrbradley commented Dec 3, 2024 • edited Loading

danielrbradley commented Dec 6, 2024

blampe commented Nov 12, 2024 •

edited

Loading

danielrbradley commented Dec 3, 2024 •

edited

Loading