Error when using S3 blobstore: java.io.IOException: failed to read entry trailer. Occured at byte: 46324 #43
Comments
|
It should look like: It should be an encrypted private key that can be decrypted using the password supplied in the passphrase field. My comment in #20 was more around "how to create and export a GPG key" (well documented by GPG) and "how to add a repo to the apt sources list" (well documented by APT), if it's not clear what should go in this field, that definitely deserves documentation! |
|
Ill try that again, but I was getting the same error. [Edit] Still the same error with just the exported private key, and even with a 2048 key. Steps: |
|
@mpoindexter I'm sorta useless here (have not made it to this section of the codebase yet). Any thing I can do to help? |
|
@DarthHater How much effort would it be for me to build nexus with a newer version of org.apache.commons commons-compress so I can see if that makes any difference (if that wont break anything)? The stack trace seems to be in the code for reading the archive and maybe not GPG related? Either that or it does not like an already signed deb possibly? I just downloaded a random deb from one of my configured PPAs to upload to test my repo. |
|
Good point that it looks like its a decompression problem! I've uploaded a lot of debs and not run into this. As far as an already signed deb, I doubt that would cause any trouble, but signed debs are pretty uncommon. Most things use repository signing (which is what this plugin does too). Is it a public deb file that you're trying to upload? If so could you link to it? If not, could you grab a random public deb (like http://archive.ubuntu.com/ubuntu/pool/main/a/a11y-profile-manager/a11y-profile-manager-doc_0.1.10-0ubuntu3_all.deb for example) and try to upload that to see if that causes trouble? |
|
Same result with that deb. I was originally trying yubikey-neo-manager_1.4.0-1_all.deb from PPA: Using: REPO_USER is user:pass assigned by user token. |
|
@ju2wheels your guess is as good as mine, I don't think it would be TOO bad. I would fork nexus-public, adjust the pom, add a |
|
Pressed wrong button PS, sorry for artificially closing this. |
|
Hmmmm, I tried uploading that package into my local copy and it worked fine. I looked at the code of commons-compress (https://github.com/apache/commons-compress/blob/master/src/main/java/org/apache/commons/compress/archivers/ar/ArArchiveInputStream.java#L125) and it seems like maybe your upload is getting truncated? So I'd look into making sure the file is downloaded properly onto your machine, and that nothing in between you and your Nexus instance is interfering with the upload (load balancers, proxies, etc), but not sure how much help I can provide beyond that given that it works on my instance. The good news is that at least it's pretty clearly a archive problem, not a problem with the GPG stuff. |
|
I tried uploading it locally on the host to 8081 but I get the same thing. That should rule out my ELB. I also tried using the Admin user's user token instead of a regular user with specific role permissions just to see if somehow I missed some permissions that might be causing the problem but that doesnt work either. @DarthHater I tried building the last commit that had 3.9.0, not too bad but failed to build both times with tests on and off. Tests on ( Tests off ( Ditto for |
|
That's wild. It's looking for a snapshot of 3.9.0, which would be gone from RSO (we are doing 3.10.0 work internally now). I'll ping a few people and let them know! |
|
I was specifically on a checkout of 3.9.0 (38a81af031551d0e9e32e474cc2d1dcd55579275), I didnt use master because my current nexus-data dir is based off 3.9.0 and I dont know what going back and forth would do the data. |
|
Gotcha, thanks for the extra info. Either way, it's looking for snapshot files rather than released ones, so that's likely the issue? I rang the bell internally, I'll see what happens. |
|
You are using this tag correct? https://github.com/sonatype/nexus-public/releases/tag/release-3.9.0-01 |
|
Thanks for the commit @ju2wheels , there's certainly something amiss with nexus-public, I'll see what we can do to figure it out. |
|
@ju2wheels I've updated the release to use the proper version and ran the build/tests locally. Give it a try now. |
|
@DarthHater @anubistheta OK I grabbed that tarball above and extracted it into a branch and got it to build successfully, however Im not convinced the produced artifact will actually run given that its missing a bunch of libraries. I updated What Im seeing is that aside from the .install4j files in the upstream official release tarball which can probably be safely ignored, there are missing libraries in the generated build (e.g. commons-compress, commons-collections4, and a few others). I dont have enough Java-fu to figure out whats going on here. Attached a file showing the diff between upstream tarball (left) and the generated tarball (right): Can be reproduced using this branch using |
|
@ju2wheels @DarthHater Thanks for the update. I'm looking into it now. |
|
@ju2wheels @DarthHater Is the upstream tarball the one download from the Sonatype website? i.e. this link: https://www.sonatype.com/download-oss-sonatype If so, I think I know what causes the discrepancy. Thus, the tarball from the website should have a superset of the features/files of the generated tarball. I think that most of the missing jar's that are in a subdir of nexus-3.9.0-01/system/com/sonatype/nexus/plugins are OSGI bundles. |
|
@DarthHater @anubistheta that would explain the missing plugins but not the other libraries unless the libraries are only included by your internal build process that includes those non OSS plugins. As far as I understand how things are packaged im not sure thats the case. For example this Apt plugin specifies a dependency on commons-compress which to my limited Java knowledge would mean the parent (nexus-public) has to provide it as part of its common installed libraries (which it currently isnt according to the diff)? At a minimum, I know these missing ones are important for this plugin: If these are not currently installed as part of the apt plugin build process, how would these get there (these are defined in If this is the expected build output then that means I cant test what I wanted to (building Nexus 3 with newer commons-compress version). Im not sure those bundle instructions are that useful in my case, im not trying to do bundle development, im just trying to rebuild a usable Nexus 3 instance with different dependency versions. |
|
@mpoindexter what version of Nexus + repository-apt are you on? |
|
I'm using 3.7.1/1.0.4 currently. But I don't think much has changed in the code aside from adding some tests and updating method signatures to match newer versions of Nexus. |
|
Interesting @ju2wheels , I see it in |
|
I just realized from looking at the file sizes and the error messages that its not truncation of the upload that seems to be the problem. Its trying to read past the end of the archive for the trailer marker (assuming that means byte sequence for EOF of archive). For example, in the error I posted above, This issue seems similar to whats being hit above except its for the TAR format instead of the AR format. Based on that discussion, could it be possible we are hitting the same problem on this line and that we arent getting a proper return value of null even though theres no next entry, so its trying to read past EOF to find a non existent next entry? |
|
@ju2wheels separately to this issue, I am digging around internally to see what people think about how to test out upgrading |
|
OK, I identified the problem based on what I found in the last post. Using an S3 blobstore with the apt repository seems to cause it to be unable to find the EOF of file I guess. Using a normal file blobstore with apt allows me to upload the deb fine, so its between S3 blob store and commons-compress. I suspect this is related to this commit that changed the temp upload from being a stream to based on the blob store. @DarthHater @mpoindexter I havent looked further but let me know if you consider this to be a bug in nexus-blobstore-s3. |
|
Great detective work @ju2wheels! I think that commit was because the TempStream based one went away in newer versions of Nexus. For the standard blob store the revised code seems to work fine, but definitely haven't tested with the s3 based one. |
|
I would raise it for s3, probably send an email to the nexus-users list and maybe create a JIRA ticket. We are in the process of bringing s3 into the core product, so I'm sure they will want to know about that. And wow, yeah! Great detective work! |
|
The interwebs seem to indicate that this could possibly be a bad interaction between the JDK's built in GZIPInputStream and streaming files from S3. If someone ever wants to look at this it might be a good experiment to replace the use of |
|
Not sure if the above change will fix this, if anyone finds this still happening with the latest code feel free to reopen! |
|
Im unfortunately hitting a convoluted mess trying to test this out to see if it resolves the issue, although nothing directly to due with this plugin. Is 3.11.0 a hard requirement for this one? Any chance it will work with 3.10.0 ? Im not able to test this in combination with S3 blobstore because that plugin wont compile against 3.11.0 . |
|
I get the same error on OSS 3.13.0-01, and the most recent apt repository plugin. Any attempts to upload a Using the default blob store works fine. This occurs via the UI, or |
|
Well, bummer. Kind of at a loss here, I don't use the S3 blobstore. As near as I can tell, the code here is right, but PRs to fix this welcome if anyone else has a good idea |
mpoindexter
changed the title
|
I'm getting something similar using the S3 blobstore. It fails with "truncated ar archive". It seems to work fine when I use local storage. |
|
I get the same issue as @jarro2783 |
|
@mpoindexter Any news on this issue? |
|
Nope. I don't use s3 blob store, don't have any way to reproduce, and I don't see anything wrong in the code. Someone who uses s3 blob store will have to look into this if it's to get fixed. |
|
Just wanted to confirm this was happening to me using S3 blobstore, switching to local one fixed it. |
|
I've finally gotten around to taking a look at this. It's a bug in |
|
APT is now part of Nexus Repository Manager. Version 3.17.0 includes the APT plugin by default. |
Thanks for creating an issue! Please fill out this form so we can be
sure to have all the information we need, and to minimize back and forth.
What are you trying to do?
Upload a deb file to an apt hosted repo.
What feature or behavior is this required for?
Upload a deb
How could we solve this issue? (Not knowing is okay!)
Im currently getting the following error:
I dont understand from the error if this is a misconfiguration on my part or a bug. It would really help if the documentation at least gave a hint as to what is wanted by the
PGP signing key pairbox despite what was said in #20 , I cant be sure im providing what it wants otherwise (e.g. if the above is actually what it wants have the documentation at least state that you should export your GPG armored public/private keys into that box or something).I gave it a RSA 4096 signing key. I get the same result even if I only provide the private key.
The text was updated successfully, but these errors were encountered: