Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

writing blob: Size mismatch #2437

Closed
edsantiago opened this issue Oct 10, 2024 · 11 comments
Closed

writing blob: Size mismatch #2437

edsantiago opened this issue Oct 10, 2024 · 11 comments
Labels
jira kind/bug A defect in an existing functionality (or a PR fixing it) stale-issue

Comments

@edsantiago
Copy link
Member

Possibly a broken image or manifest? And possibly because I ^Ced the first time I pushed it, then rebuilt it and pushed a completely different one? Anyhow I don't know the root cause nor how to investigate further nor even where to report this or if it's even a bug.

$ podman pull -q quay.io/libpod/testimage:20241009
138196c38b3baeb5bef83b1513d192fe790c856124cd3c6655b856ddfa95d867
$ skopeo copy containers-storage:quay.io/libpod/testimage:20241009 dir://tmp/sdfsdf
Getting image source signatures
Copying blob 5f70bf18a086 done   | 
Copying blob 160fb30b1281 done   | 
FATA[0000] writing blob: Size mismatch when copying sha256:160fb30b1281fc4d8c9c0bb68e63ab1b55a78be718d50f9657c0a78a030efb32, expected 11801184, got 12175360 

skopeo copy docker://quay.io/libpod/testimage:20241009 works just fine.

@edsantiago
Copy link
Member Author

Same thing with 20241010, which I did not ^C

@nalind
Copy link
Member

nalind commented Oct 10, 2024

This doesn't seem to be specific to skopeo, as I can reproduce it with podman push quay.io/libpod/testimage:20241009 dir:/tmp/sdfsdf, so I suspect it's in either containers/image or containers/storage. Adding --dest-compress to the skopeo copy command to force recompression seems to work around it.

@Luap99
Copy link
Member

Luap99 commented Oct 10, 2024

The manifest contains

"annotations": {
    "io.github.containers.compression.zstd": "true"
},

The old (currently used) testimage 20240123 does not so I assume this might have something to do with zstd compression

@mtrmac
Copy link
Contributor

mtrmac commented Oct 10, 2024

Notes:

  • The architecture-specific instance is quay.io/libpod/testimage@sha256:d58f79a56a477ac8ed3c07aa2c8f68cd05e2afdd754d40463dbf888798590aae
  • The layer is zstd:chunked.
  • 12175360 is the correct size of the payload
  • The value comes from c/storage.Layer.UncompressedSize

@mtrmac
Copy link
Contributor

mtrmac commented Oct 10, 2024

… and that size seems to come from adding the size of file contents ( chunkedDiffer.mergeTocEntries ). Yes, that’s not going to work.

@mtrmac
Copy link
Contributor

mtrmac commented Oct 10, 2024

I think I know how to fix this, working on it now.


If that hypothesis is correct, this means that re-pushes of zstd:chunked layers have been broken for about 1.5 years — i.e. we have a testing gap.

@edsantiago
Copy link
Member Author

We have many zstd testing gaps. We've never actually done any real testing anywhere in CI, only very-halfhearted efforts.

I wonder if this is the bug that keeps blocking my buildah parallel tests PR?

@mtrmac
Copy link
Contributor

mtrmac commented Oct 10, 2024

containers/storage#2130 might be a fix (absolutely untested!) to unblock, we should have a better one.


Do we have a generic: "push an image (using our default settings); pull it (without reusing any layers); push it again" test? That’s essentially what broke here.

@mtrmac
Copy link
Contributor

mtrmac commented Oct 11, 2024

Confirmed that (an updated version of) containers/storage#2130 fixes this, the metadata now contains the correct size.

@mtrmac mtrmac added the kind/bug A defect in an existing functionality (or a PR fixing it) label Oct 11, 2024
Copy link

A friendly reminder that this issue had no activity for 30 days.

@mtrmac
Copy link
Contributor

mtrmac commented Nov 22, 2024

This works correctly with skopeo-1.17.0-1.fc41.x86_64.

@mtrmac mtrmac closed this as completed Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
jira kind/bug A defect in an existing functionality (or a PR fixing it) stale-issue
Projects
None yet
Development

No branches or pull requests

5 participants