Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/build: add support for --push #4677

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 2 additions & 19 deletions cmd/buildah/push.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ import (
"github.com/containers/image/v5/manifest"
"github.com/containers/image/v5/pkg/compression"
"github.com/containers/image/v5/transports"
"github.com/containers/image/v5/transports/alltransports"
"github.com/containers/storage"
imgspecv1 "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/sirupsen/logrus"
Expand Down Expand Up @@ -148,25 +147,9 @@ func pushCmd(c *cobra.Command, args []string, iopts pushOptions) error {
return err
}

dest, err := alltransports.ParseImageName(destSpec)
// add the docker:// transport to see if they neglected it.
dest, err := util.ImageStringToImageReference(destSpec)
if err != nil {
destTransport := strings.Split(destSpec, ":")[0]
if t := transports.Get(destTransport); t != nil {
return err
}

if strings.Contains(destSpec, "://") {
return err
}

destSpec = "docker://" + destSpec
dest2, err2 := alltransports.ParseImageName(destSpec)
if err2 != nil {
return err
}
dest = dest2
logrus.Debugf("Assuming docker:// as the transport method for DESTINATION: %s", destSpec)
return fmt.Errorf("generating image reference: %w", err)
}

systemContext, err := parse.SystemContextFromOptions(c)
Expand Down
1 change: 1 addition & 0 deletions define/build.go
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ type BuildOptions struct {
// image that's meant to be run using krun as a VM instead of a conventional
// process-type container.
ConfidentialWorkload ConfidentialWorkloadOptions
Push bool
// Additional tags to add to the image that we write, if we know of a
// way to add them.
AdditionalTags []string
Expand Down
33 changes: 31 additions & 2 deletions define/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"strings"

"github.com/containers/image/v5/manifest"
imageTypes "github.com/containers/image/v5/types"
"github.com/containers/storage/pkg/archive"
"github.com/containers/storage/pkg/chrootarchive"
"github.com/containers/storage/pkg/ioutils"
Expand Down Expand Up @@ -111,9 +112,37 @@ type Secret struct {

// BuildOutputOptions contains the the outcome of parsing the value of a build --output flag
type BuildOutputOption struct {
Path string // Only valid if !IsStdout
IsDir bool
ImageRef imageTypes.ImageReference
Image string
IsStdout bool
Path string // Only valid if !IsStdout
Push bool
Type BuildOutputType

// Deprecated: Use Type instead to determine output type
IsDir bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICS IsDir is now never set. If so, keeping the field around is just as incompatible an API break as removing the field… and we might just as well remove it.

Alternatively, if we need to keep it, it must be set in a compatible way (and I don’t know what “a compatible way” would be for Type == …Image.)

Or, hypothetically, we could leave the BuildOutputOption and the existing code as is, parsing only local file outputs but keeping API compatibility, and introduce a new type and a new set of utilities that can represent image outputs.

Buildah maintainers need to decide which one of the three it is; I don’t know. (My guess is the first one, at least I found no Podman user of BuildOutputOption, but I really lack the context.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keeping the field around is just as incompatible an API break as removing the field… and we might just as well remove it.

From what I can tell, and I have very little context, the idea was to minimize disruption. Outright removing it might cause more disruption than keeping it around and not using it but I understand your concern with having dead code lying around.

}

type BuildOutputType int

const (
mtrmac marked this conversation as resolved.
Show resolved Hide resolved
danishprakash marked this conversation as resolved.
Show resolved Hide resolved
_ BuildOutputType = iota
BuildOutputImage
BuildOutputLocal
BuildOutputTar
)

// String converts a BuildOutputType into a string.
func (t BuildOutputType) String() string {
switch t {
case BuildOutputImage:
return "image"
case BuildOutputLocal:
return "local"
case BuildOutputTar:
return "tar"
}
return fmt.Sprintf("unrecognized build output type %d", t)
}

// ConfidentialWorkloadOptions encapsulates options which control whether or not
Expand Down
6 changes: 6 additions & 0 deletions docs/buildah-build.1.md
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,8 @@ Supported _keys_ are:

Valid _type_ values are:
- **local**: write the resulting build files to a directory on the client-side.
- **image**: writes the build results as an image to local storage.
- **registry**: pushes the resulting build image to the registry. Shorthand for `type=image,push=true`.
- **tar**: write the resulting files as a single tarball (.tar).

If no type is specified, the value defaults to **local**.
Expand Down Expand Up @@ -774,6 +776,10 @@ registries.conf if newer. Raise an error if a base or SBOM scanner image is
not found in the registries when image with the same name is not present
locally.

**--push**

Shorthand for "--output=type=registry"

**--quiet**, **-q**

Suppress output messages which indicate which instruction is being processed,
Expand Down
3 changes: 3 additions & 0 deletions imagebuildah/executor.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ type Executor struct {
registry string
ignoreUnrecognizedInstructions bool
quiet bool
push bool
runtime string
runtimeArgs []string
transientMounts []Mount
Expand Down Expand Up @@ -245,6 +246,7 @@ func newExecutor(logger *logrus.Logger, logPrefix string, store storage.Store, o
registry: options.Registry,
ignoreUnrecognizedInstructions: options.IgnoreUnrecognizedInstructions,
quiet: options.Quiet,
push: options.Push, // TODO: not needed if planning to update buildOutput in cli/build
runtime: options.Runtime,
runtimeArgs: options.RuntimeArgs,
transientMounts: transientMounts,
Expand Down Expand Up @@ -950,6 +952,7 @@ func (b *Executor) Build(ctx context.Context, stages imagebuilder.Stages) (image
}
}
stageID, stageRef, stageOnlyBaseImage, stageErr := b.buildStage(ctx, cleanupStages, stages, index)

if stageErr != nil {
cancel = true
ch <- Result{
Expand Down
16 changes: 13 additions & 3 deletions imagebuildah/stage_executor.go
Original file line number Diff line number Diff line change
Expand Up @@ -1268,7 +1268,7 @@ func (s *StageExecutor) Execute(ctx context.Context, base string) (imgID string,
canGenerateBuildOutput := (s.executor.buildOutput != "" && lastStage)
if canGenerateBuildOutput {
logrus.Debugf("Generating custom build output with options %q", s.executor.buildOutput)
buildOutputOption, err = parse.GetBuildOutput(s.executor.buildOutput)
buildOutputOption, err = parse.GetBuildOutput(s.executor.buildOutput, s.executor.output)
if err != nil {
return "", nil, false, fmt.Errorf("failed to parse build output: %w", err)
}
Expand Down Expand Up @@ -2363,7 +2363,17 @@ func (s *StageExecutor) commit(ctx context.Context, createdBy string, emptyLayer
return imgID, ref, nil
}

func (s *StageExecutor) generateBuildOutput(buildOutputOpts define.BuildOutputOption) error {
func (s *StageExecutor) generateBuildOutput(opts define.BuildOutputOption) error {
if opts.Type == define.BuildOutputImage {
if opts.Push {
err := internalUtil.PushImage(s.executor.store, opts, s.executor.systemContext)
if err != nil {
return fmt.Errorf("failed to export build output: %w", err)
}
}
return nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this right it does no output if not opts.Push?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iiuc, type=image outputs a container image format, only when you specify --output, it additionally pushes the image to the registry.

}

extractRootfsOpts := buildah.ExtractRootfsOptions{}
if unshare.IsRootless() {
// In order to maintain as much parity as possible
Expand All @@ -2383,7 +2393,7 @@ func (s *StageExecutor) generateBuildOutput(buildOutputOpts define.BuildOutputOp
return fmt.Errorf("failed to extract rootfs from given container image: %w", err)
}
defer rc.Close()
err = internalUtil.ExportFromReader(rc, buildOutputOpts)
err = internalUtil.ExportFromReader(rc, opts)
if err != nil {
return fmt.Errorf("failed to export build output: %w", err)
}
Expand Down
33 changes: 31 additions & 2 deletions internal/util/util.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
package util

import (
"context"
"fmt"
"io"
"os"
"path/filepath"

"github.com/containers/buildah/define"
"github.com/containers/buildah/util"
"github.com/containers/common/libimage"
lplatform "github.com/containers/common/libimage/platform"
"github.com/containers/image/v5/types"
Expand Down Expand Up @@ -50,6 +52,29 @@ func NormalizePlatform(platform v1.Platform) v1.Platform {
}
}

// PushImage copies contents of the image to a new location
func PushImage(store storage.Store, opts define.BuildOutputOption, systemCtx *types.SystemContext) error {
runtime, err := libimage.RuntimeFromStore(store, &libimage.RuntimeOptions{SystemContext: systemCtx})
if err != nil {
return err
}

imageRef, err := util.ImageStringToImageReference(opts.Image)
if err != nil {
return fmt.Errorf("failed to convert image to ImageReference")
}

libimageOptions := &libimage.PushOptions{}
libimageOptions.Writer = os.Stdout
dest := fmt.Sprintf("%s:%s", imageRef.Transport().Name(), imageRef.StringWithinTransport())
_, err = runtime.Push(context.Background(), opts.Image, dest, libimageOptions)
if err != nil {
return fmt.Errorf("failed while pushing image %+q: %w", opts.ImageRef, err)
}

return nil
}

// ExportFromReader reads bytes from given reader and exports to external tar, directory or stdout.
func ExportFromReader(input io.Reader, opts define.BuildOutputOption) error {
var err error
Expand All @@ -59,7 +84,8 @@ func ExportFromReader(input io.Reader, opts define.BuildOutputOption) error {
return err
}
}
if opts.IsDir {
switch opts.Type {
case define.BuildOutputLocal:
danishprakash marked this conversation as resolved.
Show resolved Hide resolved
// In order to keep this feature as close as possible to
// buildkit it was decided to preserve ownership when
// invoked as root since caller already has access to artifacts
Expand All @@ -81,7 +107,7 @@ func ExportFromReader(input io.Reader, opts define.BuildOutputOption) error {
if err != nil {
return fmt.Errorf("failed while performing untar at %q: %w", opts.Path, err)
}
} else {
case define.BuildOutputTar:
outFile := os.Stdout
if !opts.IsStdout {
outFile, err = os.Create(opts.Path)
Expand All @@ -94,7 +120,10 @@ func ExportFromReader(input io.Reader, opts define.BuildOutputOption) error {
if err != nil {
return fmt.Errorf("failed while performing copy to %q: %w", opts.Path, err)
}
default:
return fmt.Errorf("build output type %s not supported", opts.Type)
}

return nil
}

Expand Down
12 changes: 11 additions & 1 deletion pkg/cli/build.go
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ func GenBuildOptions(c *cobra.Command, inputArgs []string, iopts BuildOptions) (
timestamp = &t
}
if c.Flag("output").Changed {
buildOption, err := parse.GetBuildOutput(iopts.BuildOutput)
buildOption, err := parse.GetBuildOutput(iopts.BuildOutput, output)
if err != nil {
return options, nil, nil, err
}
Expand All @@ -280,6 +280,15 @@ func GenBuildOptions(c *cobra.Command, inputArgs []string, iopts BuildOptions) (
return options, nil, nil, err
}
}

if c.Flag("push").Changed {
if len(iopts.BuildOutput) == 0 {
danishprakash marked this conversation as resolved.
Show resolved Hide resolved
iopts.BuildOutput = "type=registry"
} else {
return options, nil, nil, fmt.Errorf("cannot set both --push and --output")
}
}

var cacheTo []reference.Named
var cacheFrom []reference.Named
cacheTo = nil
Expand Down Expand Up @@ -399,6 +408,7 @@ func GenBuildOptions(c *cobra.Command, inputArgs []string, iopts BuildOptions) (
OutputFormat: format,
Platforms: platforms,
PullPolicy: pullPolicy,
Push: iopts.Push,
Quiet: iopts.Quiet,
RemoveIntermediateCtrs: iopts.Rm,
ReportWriter: reporter,
Expand Down
2 changes: 2 additions & 0 deletions pkg/cli/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ type BudResults struct {
Pull string
PullAlways bool
PullNever bool
Push bool
Quiet bool
IdentityLabel bool
Rm bool
Expand Down Expand Up @@ -305,6 +306,7 @@ newer: only pull base and SBOM scanner images when newer images exist on the r
fs.BoolVar(&flags.Stdin, "stdin", false, "pass stdin into containers")
fs.StringArrayVarP(&flags.Tag, "tag", "t", []string{}, "tagged `name` to apply to the built image")
fs.StringVarP(&flags.BuildOutput, "output", "o", "", "output destination (format: type=local,dest=path)")
fs.BoolVar(&flags.Push, "push", false, "Shorthand for `--output=type=registry`")
danishprakash marked this conversation as resolved.
Show resolved Hide resolved
danishprakash marked this conversation as resolved.
Show resolved Hide resolved
fs.StringVar(&flags.Target, "target", "", "set the target build stage to build")
fs.Int64Var(&flags.Timestamp, "timestamp", 0, "set created timestamp to the specified epoch seconds to allow for deterministic builds, defaults to current time")
fs.BoolVar(&flags.TLSVerify, "tls-verify", true, "require HTTPS and verify certificates when accessing the registry")
Expand Down
Loading
Loading