[bug] verify base image pre-submit fails

[laurentsimon]

See https://github.com/slsa-framework/slsa-github-generator/actions/runs/3751444140/jobs/6372431280

I have temporarily disabled this check in the branch protection setting

[ianlewis]

It seems like they re-build and push all the old image tags when they add a new image tag. It's really annoying because it makes it hard to know if an image has been modified out from under us, but I think updating the sha should fix it.

[ianlewis]

Maybe we don't really need to verify the sha if we can verify the image at the tag?

[laurentsimon]

It seems like they re-build and push all the old image tags when they add a new image tag. It's really annoying because it makes it hard to know if an image has been modified out from under us, but I think updating the sha should fix it.

So you're saying they re-build (non-deterministically) and the hash changes?

@loosebazooka any insights?

[ianlewis]

Yeah, I'm pretty sure most docker builds aren't hermetic/deterministic.

I filed a bug in the repo for the official golang image so we can see what they say as well.
docker-library/golang#448

[ianlewis]

I updated the sha in #1322 and re-added verify base images as required check so that should fix the issue in the short term.

[loosebazooka]

There are ways to do reproducible builds. For example with ko and a distroless base image.

Oh the golang image?

[ianlewis]

There are ways to do reproducible builds. For example with ko and a distroless base image.

Oh the golang image?

Yeah, this is the official golang image. They literally push new images for every tag all the time. I literally just fixed it and they pushed the image again and updated the tags 40 min ago (v annoying). I guess they just assume that folks won't reference images by sha and update their tags constantly.

[ianlewis]

We pin the base image by sha and verify Docker content trust on the image, but since they update the tags all the time the sha that we expect changes and our image verification script fails.

I guess we could just stop pinning the base image by sha because they clearly don't expect us to and it's pretty useless to do so if they keep updating it.

[loosebazooka]

Oh, those are built by docker, not the golang team. Perhaps @Sajmani knows where the official golang team images are?

[jonjohnsonjr]

They rebuild the tags to pick up security fixes. This is expected. Tags in container registries are more like git branches than git tags.

[ianlewis]

They rebuild the tags to pick up security fixes. This is expected. Tags in container registries are more like git branches than git tags.

I kind of understand that better now and I can understand some of the benefits like security updates etc. Thanks. It does make it tricky to work with DCT though.

Right now we are using a Docker container action which builds the image on ever GHA run. It would pick up the changes in the base image tag immediately and AFAIK has no way of verifying the images before using it.

So we have a couple options:

1. Don't pin by sha. This means that any changes to the base image tag would get picked up immediately and we wouldn't have a way to verify that the base images we are using are verified images. So I think this is out.
2. Switch to using a pre-built Docker image and verify DCT of base images when we build the image. We can then pin by sha to make sure it hasn't changed since we verified it. This might be the most practical answer, short term, but does require a decent amount of work to set up building, and managing the image.
3. Rewrite our action to use TS or something else and don't use Docker container actions. Long term, I think we want to do this anyway.

Not sure whether #2 or #3 is going to be more work in the end.