How to mirror maven repository without the maven2/ part of the URL

[miller79]

Following the guidance on setting up a mirror for Packeto from here, the mapping only allows you to replace the "HOST_URL" part of the path. With the maven repository, the URL pattern is typically https://repo1.maven.org/maven2/path-to-file. With this, the mirrors will typically replace the entire https://repo1.maven.org/maven2/ portion of the URL. There doesn't seem to be a way to do this replacement and having our mirrored repository contain the maven2 would be a bit of pain on our side.

I'm curious if anyone else has ran into this issue and what you have done to get around it.

[dmikusa]

@loewenstein @bitgully @shresthaujjwal ^^^

[loewenstein-sap]

Yeah, I think we've seen the same problem and didn't find a solution either.

cc @c0d1ngm0nk3y @pbusko @modulo11 @nicolasbender

[bitgully]

Should we introduce an additional feature to skip starting parts of the path from the original URL?
Maybe by not only replacing the hostname but also the path that follows, according to the most specific match.
For example:

/platform
    └── bindings
        └── dependency-mirror
            ├── default                     https://mirror.example.org
            ├── repo1.maven.org%2Fmaven2    https://maven-mirror.example.org
            └── type                        dependency-mirror

Here, https://repo1.maven.org/maven2/path-to-file would translate to https://maven-mirror.example.org/path-to-file. Whilst https://repo1.maven.org/maven3/path-to-file would translate to https://mirror.example.org/maven3/path-to-file because there is no more specific match available.
The same mirror URLs can be used for both definitions, essentially just dropping the /maven2 part

[miller79]

This seems reasonable to me and when using the BP_DEPENDENCY_MIRROR environment variable, I can see updating the rules to be something like the following:

"Dots (.) of the original hostname must be replaced with a single underscore (_), dashes (-) are replaced with a double underscore (__), and slashes (/) are replaced with a triple underscore (___)."

[dmikusa]

Yes, I think this adds some complexity and I'd like to avoid this option if possible. There is added complexity in terms of what keys match, but also how to apply the value. If there's no path on the value, does it mean you want the path stripped or does it mean you want the full path? I don't think there's enough information in the proposed set up to know specifically what to do in those situations, I think we'd need to add more metadata to break some of these edge cases, and that's where the complexity comes in.

[bitgully]

I'm not so happy with my first suggestion as this would reduce usability and make things more cryptic.
Maybe it would make more sense to add a variable like BP_DEPENDENCY_MIRROR_SKIP_PATHS.
The value of this could hold a list like /maven2,/other1/sub-other1,/other2. In the case reported, we would only need the /maven2 value to be set.

Any thoughts on this?

[dmikusa]

I think a separate config setting is going to be needed, that way it's opt-in. If you don't set it, then we're not adding any UX complexity for users. Only users and repos that need the additional configureability have to deal with the additional complexity.

How would BP_DEPENDENCY_MIRROR_SKIP_PATHS apply for multiple repo mappings though? Would all of these apply to all of the repos? I could see that causing issues if you had one repo where the path should apply, and another where it shouldn't apply. It seems like we probably need to have something that is more specific, like to the repo in question.

What about BP_DEPENDENCY_MIRROR_REPO1_MAVEN_ORG_SKIP_PATH=/maven2? We could have a similar setting BP_DEPENDENCY_MIRROR_SKIP_PATH=/maven2 if you want to apply that to the default mirror.

PROS:

1. We're following the same configuration convention that we use for mapping a specific repository to a specific mirror, so it's familiar to users.
2. It only impacts mirrors that need this, not all users
3. It's specific, you know exactly what it'll match. Plus, it matches the same way that we match dependency mirrors.
4. Most importantly, we're not requiring the use of regex :)

CONS:

1. It's verbose, but I'm not sure there's a way around that.

For bindings that would give us something like this...

/platform
    └── bindings
        └── dependency-mirror
            ├── default                     https://mirror.example.org
            ├── repo1.maven.org    https://maven-mirror.example.org
            └── type                        dependency-mirror
        └── dependency-mirror-skip-path
            ├── repo1.maven.org    /maven2
            └── type                        dependency-mirror-skip-path

I'm not a huge fan of this cause it's verbose and duplicates the keys, but it's simple (to create & parse), so I could probably live with it if others agree on it as the path forward.

I could potentially see using some encoded format to make it more brief, like JSON or maybe something simpler URL request query parameters. We could have it behave such that the encoding is not required unless you need more configuration.

/platform
    └── bindings
        └── dependency-mirror
            ├── default                     https://mirror.example.org
            ├── repo1.maven.org    mirror=https://maven-mirror.example.org,skip-path=/maven2
            └── type                        dependency-mirror

Thoughts?

[miller79]

From a binding prospective, I really like @dmikusa second suggestion of having mirror and skip-path inside of the mirrored option. But I'm I don't see how something like this would translate to the BP_DEPENDENCY_MIRROR syntax. I still think at least from that prospective UX wise its not confusing to add another underscore (___) to represent the slash in the path which is a "soft" opt-in approach. But with the conflicting methods, makes me feel like this one probably isn't a keeper.

However, if the first option contains the way people are comfortable with currently with an additional way, then I feel the BP_DEPENDECY_MIRROR_SKIP_PATH makes sense to me but I would personally prefer the SKIP_PATH part is before to help imply its the property with a value after it so it could also be the default value. And I think the url duplication isn't that big of a problem.

[bitgully]

Since Maven itself is so popular, we should keep the extra configuration regarding the /maven2 path to a minimum as most users will probably have to skip this.

I would be happy with @dmikusa's last suggestion using the optional encoding. This seems to be a good compromise between simplicity and setting skip-paths in a host-specific manner. The encoded string with additional arguments can also be set as the env variable's value, removing the need for additional variables (and the triple underscore notation).

Whilst less powerful, the plain comma separated encoding might be more user-friendly than JSON, due to the need for escaping the double quotes when setting the variable.

[dmikusa]

Ok, so sounds like we're leaning toward the second option, encoding it in the existing configuration methods.

Examples:

• BP_DEPENDENCY_MIRROR=https://mirror.example.org,skip-path=/foo
• BP_DEPENDENCY_MIRROR_REPO1_MAVEN_ORG=https://mirror.example.org,skip-path=/maven2

or

/platform
    └── bindings
        └── dependency-mirror
            ├── default                     https://mirror.example.org
            ├── repo1.maven.org    mirror=https://maven-mirror.example.org,skip-path=/maven2
            └── type                        dependency-mirror

There is always the possibility that a URL have a comma in it. If that is case with your URL then it should be URL encoded so that we can properly parse things.

Ex: BP_DEPENDENCY_MIRROR_REPO1_MAVEN_ORG=https://mirror.example.org/buildpack%2Cdependencies,skip-path=/maven2

Let's get an agreement. Please 👍 or 👎 this comment. Thanks

[bitgully]

An implementation of the "skip-path" feature has been proposed for the libpak library in paketo-buildpacks/libpak#366. If this is found reasonable, a similar enhancement should be added in https://github.com/paketo-buildpacks/packit.

[shresthaujjwal]

@bitgully thank you so much for enhancing the dependency mapping. Sorry for responding late had some other priorities last few month.

This mapping work great for maven download if you have a mirror and only thing you have to skip is the path but I don't think it will work for us for gradle download as the mirror for that is setup differently. For example public download path for gradle is

https://downloads.gradle.org/distributions/**

But internally our mirror is setup like this

https://mirror.internal.com/subpath/gradle-distributions-cache/**

Thoughts on this pattern

[shresthaujjwal]

Is it possible to enhance this more and add another parameter that does replace. Just like skip-path we have replace-path too. Just a thought

[miller79]

Just talking it out but couldn't you just set your example as the following?

BP_DEPENDENCY_MIRROR_DOWNLOADS_GRADLE_ORG=https://mirror.internal.com/subpath/gradle-distributions-cache,skip-path=/distributions

If I'm understanding your use case, then I believe this will work for you as well.

[shresthaujjwal]

ah yes @miller79 you are right didn't think about that. Looking fw to test this out when its ready

[bitgully]

Thanks @dmikusa for your review and merge into libpak. I've now ported the implementation to packit (currently awaiting review).