Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use new docker build provider #1278

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Use new docker build provider #1278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8859481
docker-build
blampe Apr 26, 2024
1848695
fix dependencies
blampe Aug 7, 2024
c1c5778
regenerate schema + sdk
blampe Aug 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 36 additions & 21 deletions awsx/ecr/image.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
// limitations under the License.

import * as aws from "@pulumi/aws";
import * as docker from "@pulumi/docker";
import * as docker from "@pulumi/docker-build";
import * as pulumi from "@pulumi/pulumi";
import * as schema from "../schema-types";
import * as utils from "../utils";
Expand Down Expand Up @@ -62,37 +62,52 @@ export function computeImageFromAsset(
throw new Error("Invalid credentials");
}
return {
registry: ecrCredentials.proxyEndpoint,
address: ecrCredentials.proxyEndpoint,
username: username,
password: password,
};
});

let cacheFrom: docker.types.input.CacheFromArgs[] = [];
if (dockerInputs.cacheFrom !== undefined) {
cacheFrom = dockerInputs.cacheFrom.map((c) => {
return {
registry: {
ref: c,
},
};
});
}
// Use an inline cache by default.
if (cacheFrom.length === 0) {
cacheFrom.push({ registry: { ref: canonicalImageName } });
}

let context = ".";
if (dockerInputs.context !== undefined) {
context = dockerInputs.context;
}

const dockerImageArgs: docker.ImageArgs = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any new input properties we should expose as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably! There's a lot more we can do now, e.g. multiple tags, multiple platforms, more refined caching behavior, etc.

I didn't want to add any of those right now because (a) these are opinionated APIs and opinions take more time to get concensus on, and (b) it's not clear how much (if any) we want to continue investing in this wrapper versus giving users more composable building blocks (like your example of registry auth helpers). The latter is my preference, but it's not my call to make.

imageName: canonicalImageName,
build: {
args: dockerInputs.args,
builderVersion: dockerInputs.builderVersion,
cacheFrom: dockerInputs.cacheFrom
? {
images: dockerInputs.cacheFrom,
}
: undefined,
context: dockerInputs.context,
dockerfile: dockerInputs.dockerfile,
platform: dockerInputs.platform,
target: dockerInputs.target,
},
registry: registryCredentials,
tags: [canonicalImageName],
buildArgs: dockerInputs.args,
cacheFrom: cacheFrom,
cacheTo: [{ inline: {} }],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if this will increase image sizes substantially? IIRC this is only metadata about the layers so the size increase shouldn't be substantial.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great question! It's actually full cache artifacts, so it can increase the size a bit. I think it would be best to take this out and stay as close to the existing behavior of not doing any caching stuff unless cacheFrom is set.

However this makes me realize we probably do want inline caching when cacheFrom is set. Currently it's using a registry cache output, but this isn't compatible with the existing behavior because users currently point this at an image (not a cache artifact) due to the way the old Docker provider did caching. (It would simply pull images to get layers into your local build cache.) In other words we don't want to export a cache to image:latest, but it would be reasonable to push an image with inline caching.

Caching Docker images is hard enough as it is and the old provider's behavior was also easy to get wrong. For example cacheFrom might not do anything without a magic env var:

new docker.Image("my-app-image", {
    build: {
        args: {
            BUILDKIT_INLINE_CACHE: "1",
        },
        cacheFrom: {
            images: ["foo:latest"],
        },
        context: "app/",
        dockerfile: "app/Dockerfile",
    },
    imageName: "foo:latest",
});

So for us to stay as close to the existing behavior while also allowing caching to somewhat work, I think we basically just want to enable this BUILDKIT_INLINE_CACHE behavior when cacheFrom is present.

context: { location: context },
dockerfile: { location: dockerInputs.dockerfile },
platforms: dockerInputs.platform ? [dockerInputs.platform as docker.Platform] : [],
target: dockerInputs.target,
push: true,
registries: [registryCredentials],
};

const image = new docker.Image(imageName, dockerImageArgs, { parent });

image.repoDigest.apply((d: any) =>
pulumi.log.debug(` build complete: ${imageName} (${d})`, parent),
);
image.ref.apply((ref) => {
pulumi.log.debug(` build complete: ${ref}`, parent);
});

return image.repoDigest;
return image.ref;
}

function createUniqueImageName(inputs: pulumi.Unwrap<schema.DockerBuildInputs>): string {
Expand Down
2 changes: 1 addition & 1 deletion awsx/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"//": "Pulumi sub-provider dependencies must be pinned at an exact version because we extract this value to generate the correct dependency in the schema",
"dependencies": {
"@pulumi/aws": "6.47.0",
"@pulumi/docker": "4.5.1",
"@pulumi/docker-build": "0.0.5",
"@pulumi/pulumi": "3.127.0",
"@types/aws-lambda": "^8.10.23",
"docker-classic": "npm:@pulumi/[email protected]",
Expand Down
1 change: 0 additions & 1 deletion awsx/schema-types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ export type Functions = {
"awsx:ec2:getDefaultVpc": (inputs: getDefaultVpcInputs) => Promise<getDefaultVpcOutputs>;
};
import * as aws from "@pulumi/aws";
import * as docker from "@pulumi/docker";
export abstract class Trail<TData = any> extends (pulumi.ComponentResource)<TData> {
public bucket?: aws.s3.Bucket | pulumi.Output<aws.s3.Bucket>;
public logGroup?: aws.cloudwatch.LogGroup | pulumi.Output<aws.cloudwatch.LogGroup>;
Expand Down
1 change: 0 additions & 1 deletion awsx/scripts/generate-provider-types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ const externalRefs = (() => {
};
};
addRef("aws");
addRef("docker");
return externalRefs;
})();

Expand Down
9 changes: 4 additions & 5 deletions awsx/yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1666,13 +1666,12 @@
mime "^2.0.0"
resolve "^1.7.1"

"@pulumi/docker@4.5.1":
version "4.5.1"
resolved "https://registry.yarnpkg.com/@pulumi/docker/-/docker-4.5.1.tgz#9058851bebbb358a1081c765d928fd6791c6c7ba"
integrity sha512-2BTFycFLwSpHGQ4IFTsUHl8H5w81AgkrMHSLUQ8Zu6HBDgGhB5up6YsxVqLeaUeWAedEUrrSCY3xTCNbP4a0ag==
"@pulumi/docker[email protected]":
version "0.0.5"
resolved "https://registry.yarnpkg.com/@pulumi/docker-build/-/docker-build-0.0.5.tgz#9e86ac0761b7fba4f24064095658bf4cf3a03a42"
integrity sha512-oaPSvgwQ0FclGKz8WGdlPvSV4Iw1rg7HkL6dqrTRxNpXBnlPRodDuYRyk/hWCrvQx+dZchJna4j3urFXOWPCEw==
dependencies:
"@pulumi/pulumi" "^3.0.0"
semver "^5.4.0"

"@pulumi/[email protected]", "@pulumi/pulumi@^3.0.0":
version "3.127.0"
Expand Down
8 changes: 4 additions & 4 deletions schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,22 +27,22 @@
"generateResourceContainerTypes": true,
"importBasePath": "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx",
"internalDependencies": [
"github.com/pulumi/pulumi-docker/sdk/v4/go/docker"
"github.com/pulumi/pulumi-docker-build/sdk/go/dockerbuild"
],
"liftSingleValueMethodReturns": true,
"respectSchemaVersion": true
},
"java": {
"dependencies": {
"com.pulumi:aws": "6.47.0",
"com.pulumi:docker": "4.5.1"
"com.pulumi:docker-build": "0.0.5"
}
},
"nodejs": {
"dependencies": {
"@aws-sdk/client-ecs": "^3.405.0",
"@pulumi/aws": "^6.47.0",
"@pulumi/docker": "^4.5.1",
"@pulumi/docker-build": "^0.0.5",
"@pulumi/pulumi": "^3.0.0",
"@types/aws-lambda": "^8.10.23",
"aws-sdk": "^2.1450.0",
Expand All @@ -65,7 +65,7 @@
"requires": {
"pulumi": "\u003e=3.91.1,\u003c4.0.0",
"pulumi-aws": "\u003e=6.0.4,\u003c7.0.0",
"pulumi-docker": "\u003e=4.5.1,\u003c5.0.0"
"pulumi-docker-build": "\u003e=0.0.5,\u003c1.0.0"
},
"respectSchemaVersion": true,
"usesIOClasses": true
Expand Down
32 changes: 16 additions & 16 deletions schemagen/pkg/gen/schema.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ func GenerateSchema(packageDir string) schema.PackageSpec {
dependencies := readPackageDependencies(packageDir)
awsSpec := getPackageSpec("aws", dependencies.Aws)
awsNativeSpec := getPackageSpec("aws-native", awsNativeTypesVersion)
dockerSpec := getPackageSpec("docker", dependencies.Docker)
dockerSpec := getPackageSpec("docker-build", dependencies.Docker)

packageSpec := schema.PackageSpec{
Name: "awsx",
Expand Down Expand Up @@ -67,25 +67,25 @@ func GenerateSchema(packageDir string) schema.PackageSpec {
"generateResourceContainerTypes": true,
"importBasePath": "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx",
"liftSingleValueMethodReturns": true,
"internalDependencies": []string{"github.com/pulumi/pulumi-docker/sdk/v4/go/docker"},
"internalDependencies": []string{"github.com/pulumi/pulumi-docker-build/sdk/go/dockerbuild"},
"respectSchemaVersion": true,
}),
"java": rawMessage(map[string]interface{}{
"dependencies": map[string]string{
"com.pulumi:aws": dependencies.Aws,
"com.pulumi:docker": dependencies.Docker,
"com.pulumi:aws": dependencies.Aws,
"com.pulumi:docker-build": dependencies.Docker,
},
}),
"nodejs": rawMessage(map[string]interface{}{
"dependencies": map[string]string{
"@aws-sdk/client-ecs": "^3.405.0",
"@pulumi/pulumi": "^3.0.0",
"@pulumi/aws": "^" + dependencies.Aws,
"@pulumi/docker": "^" + dependencies.Docker,
"docker-classic": "npm:@pulumi/[email protected]",
"@types/aws-lambda": "^8.10.23",
"aws-sdk": "^2.1450.0",
"mime": "^2.0.0",
"@aws-sdk/client-ecs": "^3.405.0",
"@pulumi/pulumi": "^3.0.0",
"@pulumi/aws": "^" + dependencies.Aws,
"@pulumi/docker-build": "^" + dependencies.Docker,
"docker-classic": "npm:@pulumi/[email protected]",
"@types/aws-lambda": "^8.10.23",
"aws-sdk": "^2.1450.0",
"mime": "^2.0.0",
},
"devDependencies": map[string]string{
"@types/node": "^18",
Expand All @@ -96,9 +96,9 @@ func GenerateSchema(packageDir string) schema.PackageSpec {
}),
"python": rawMessage(map[string]interface{}{
"requires": map[string]string{
"pulumi": ">=3.91.1,<4.0.0",
"pulumi-aws": ">=6.0.4,<7.0.0",
"pulumi-docker": fmt.Sprintf(">=%s,<5.0.0", dependencies.Docker),
"pulumi": ">=3.91.1,<4.0.0",
"pulumi-aws": ">=6.0.4,<7.0.0",
"pulumi-docker-build": fmt.Sprintf(">=%s,<1.0.0", dependencies.Docker),
},
"usesIOClasses": true,
"readme": "Pulumi Amazon Web Services (AWS) AWSX Components.",
Expand Down Expand Up @@ -256,7 +256,7 @@ func rawMessage(v interface{}) schema.RawMessage {

type Dependencies struct {
Aws string `json:"@pulumi/aws"`
Docker string `json:"@pulumi/docker"`
Docker string `json:"@pulumi/docker-build"`
Pulumi string `json:"@pulumi/pulumi"`
}

Expand Down
15 changes: 7 additions & 8 deletions sdk/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ toolchain go1.22.4
require (
github.com/blang/semver v3.5.1+incompatible
github.com/pulumi/pulumi-aws/sdk/v6 v6.47.0
github.com/pulumi/pulumi-docker/sdk/v4 v4.4.3
github.com/pulumi/pulumi-docker-build/sdk/go/dockerbuild v0.0.3
github.com/pulumi/pulumi/sdk/v3 v3.127.0
)

Expand All @@ -18,15 +18,15 @@ require (
github.com/ProtonMail/go-crypto v1.0.0 // indirect
github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
github.com/agext/levenshtein v1.2.3 // indirect
github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/charmbracelet/bubbles v0.16.1 // indirect
github.com/charmbracelet/bubbletea v0.25.0 // indirect
github.com/charmbracelet/lipgloss v0.7.1 // indirect
github.com/cheggaaa/pb v1.0.29 // indirect
github.com/cloudflare/circl v1.3.7 // indirect
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
github.com/containerd/console v1.0.4 // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/djherbis/times v1.5.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
Expand All @@ -40,7 +40,7 @@ require (
github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/hcl/v2 v2.17.0 // indirect
github.com/hashicorp/hcl/v2 v2.19.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
Expand All @@ -65,19 +65,18 @@ require (
github.com/rivo/uniseg v0.4.4 // indirect
github.com/rogpeppe/go-internal v1.12.0 // indirect
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/skeema/knownhosts v1.2.2 // indirect
github.com/spf13/cobra v1.8.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stretchr/objx v0.5.0 // indirect
github.com/texttheater/golang-levenshtein v1.0.1 // indirect
github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/zclconf/go-cty v1.13.2 // indirect
go.uber.org/atomic v1.9.0 // indirect
github.com/zclconf/go-cty v1.14.1 // indirect
go.uber.org/atomic v1.10.0 // indirect
golang.org/x/crypto v0.24.0 // indirect
golang.org/x/exp v0.0.0-20240604190554-fc45aab8b7f8 // indirect
golang.org/x/mod v0.18.0 // indirect
Expand Down
Loading
Loading