Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exclude httpclientjava #81

Closed
jonahgraham opened this issue Nov 14, 2023 · 18 comments · Fixed by #82 or #87
Closed

Exclude httpclientjava #81

jonahgraham opened this issue Nov 14, 2023 · 18 comments · Fixed by #82 or #87

Comments

@jonahgraham
Copy link
Contributor

Because the Eclipse p2 changed to use a new ECF provider that is known to not work for some consumers we need to exclude that new provider in EPP by adding:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava

For additional details see eclipse-equinox/p2#381 and N&N entry

cc: @merks
jonahgraham added a commit to jonahgraham/simrel.build that referenced this issue Nov 14, 2023
@jonahgraham
Include ecf.filetransfer.httpclient5 in SimRel
7c4a957 
This is to workaround:

- eclipse-equinox/p2#381

And to enable:

- eclipse-packaging/packages#81

in 2023-12 M3.

The Eclipse Platform is expected to contribute this for RC1
but including this from ECF allows us to start testing this
in 2023-12 M3
@jonahgraham jonahgraham mentioned this issue Nov 14, 2023
Merged
@merks
Copy link
Contributor

merks commented Nov 14, 2023

I’ve confirmed via tracing in the installer that this works.

Also a user has confirmed that the installer works again in their network

eclipse-oomph/oomph#46

Of course this should be done for rc1 not m3.

@merks
Copy link
Contributor

merks commented Nov 14, 2023

Of course if you include the old provider in the products for m3 then doing this for m3 is fine and good...

jonahgraham added a commit to eclipse-simrel/simrel.build that referenced this issue Nov 14, 2023
@jonahgraham
Include ecf.filetransfer.httpclient5 in SimRel (#57)
5dffc95 
This is to workaround:

- eclipse-equinox/p2#381

And to enable:

- eclipse-packaging/packages#81

in 2023-12 M3.

The Eclipse Platform is expected to contribute this for RC1
but including this from ECF allows us to start testing this
in 2023-12 M3
@jonahgraham jonahgraham mentioned this issue Nov 16, 2023
Merged
@jonahgraham jonahgraham reopened this Nov 16, 2023
@jonahgraham
Copy link
Contributor Author

Something has gone wrong - I didn't get the same results on the build machine as I got when building locally and the produced builds are not working. I tested locally with the rcp package. Best I can tell because of this change.

I get runtime wiring errors related to httpclient, for rcp they seem to be relatively minor (broken, but not too much broken), but for cpp it is more catastrophic.

I may have to roll back this fix for M3 and keep trying.

@merks
Copy link
Contributor

merks commented Nov 16, 2023

Is MPC’s stale dependency on older versions a problem?

jonahgraham added a commit to jonahgraham/packages that referenced this issue Nov 16, 2023
@jonahgraham
Revert "Use Apache HTTP Client by default in EPP"
654f390 
This seems to have caused p2 to make an application solution
that doesn't work.

See eclipse-packaging#81

This reverts commit 3083907.
@jonahgraham jonahgraham mentioned this issue Nov 16, 2023
Merged
@jonahgraham
Copy link
Contributor Author

I'm running a build without the change to make sure I have something for M3 today while I try to find the cause.

@jonahgraham
Copy link
Contributor Author

Is MPC’s stale dependency on older versions a problem?

Yes - I think so. I'm away from my main dev computer at the moment trying to solve this. I will post a full error message soon.

@jonahgraham
Copy link
Contributor Author

Here are some of the wiring errors:


!ENTRY org.eclipse.epp.mpc.core 4 0 2023-11-16 15:04:11.456
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.epp.mpc.core [228]
  Unresolved requirement: Import-Package: org.apache.hc.client5.http; version="[5.1.0,5.2.0)"


!ENTRY org.eclipse.epp.mpc.core.win32 4 0 2023-11-16 15:04:11.456
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.epp.mpc.core.win32 [229]
  Unresolved requirement: Import-Package: org.apache.hc.client5.http.auth; version="[5.1.0,5.2.0)"

I had many more errors on my Linux build and I will retry with that later.

@merks
Copy link
Contributor

merks commented Nov 16, 2023

I noticed that looking at duplicates. I didn’t consider it might cause wiring problems. It’s beyond frustrating that 1/2 the projects just don’t bother to provide updates and have crazy restrictive ranges.

@jonahgraham
Copy link
Contributor Author

I tried to "just" include the same feature that Eclipse Platform will be delivering in RC1 - so I guess RC1 from Platform won't solve the problem and we'll be in the same boat.

@merks
Copy link
Contributor

merks commented Nov 16, 2023

I will test the egg laying wool milk pig tomorrow which I expect will have the same problem. Then I will need to poke MPC to update their contribution ASAP. It’s beyond annoying…

@jonahgraham
Copy link
Contributor Author

jonahgraham commented Nov 16, 2023 •
edited
Loading

It looks like when Tycho/p2 pulled in httpclient 5.1 and 5.2. And with both of them installed we end up with uses violations:

  Bundle was not resolved because of a uses constraint violation.
  org.apache.felix.resolver.reason.ReasonException: Uses constraint violation. Unable to resolve resource org.eclipse.epp.mpc.core [osgi.identity; type="osgi.bundle"; version:Version="1.10.1.v20221110-1841"; osgi.identity="org.eclipse.epp.mpc.core"; singleton:="true"] because it is exposed to package 'org.apache.hc.client5.http' from resources org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.1.3.v20221013-1742"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"] and org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.2.1.v20230802-0806"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"] via two dependency chains.

Chain 1:
  org.eclipse.epp.mpc.core [osgi.identity; type="osgi.bundle"; version:Version="1.10.1.v20221110-1841"; osgi.identity="org.eclipse.epp.mpc.core"; singleton:="true"]
    import: (&(osgi.wiring.package=org.apache.hc.client5.http)(&(version>=5.1.0)(!(version>=5.2.0))))
     |
    export: osgi.wiring.package: org.apache.hc.client5.http
  org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.1.3.v20221013-1742"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"]

Chain 2:
  org.eclipse.epp.mpc.core [osgi.identity; type="osgi.bundle"; version:Version="1.10.1.v20221110-1841"; osgi.identity="org.eclipse.epp.mpc.core"; singleton:="true"]
    import: (&(osgi.wiring.package=org.apache.hc.client5.http.impl.classic)(&(version>=5.1.0)(!(version>=5.2.0))))
     |
    export: osgi.wiring.package=org.apache.hc.client5.http.impl.classic; uses:=org.apache.hc.client5.http.routing
  org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.1.3.v20221013-1742"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"]
    import: (&(osgi.wiring.package=org.apache.hc.client5.http.routing)(&(version>=5.1.0)(!(version>=6.0.0))))
     |
    export: osgi.wiring.package: org.apache.hc.client5.http.routing; uses:=org.apache.hc.client5.http
    export: osgi.wiring.package=org.apache.hc.client5.http
  org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.2.1.v20230802-0806"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"]
        at org.eclipse.osgi.container.Module.start(Module.java:463)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1852)
        at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1845)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1786)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1750)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1672)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)

@merks
Copy link
Contributor

merks commented Nov 16, 2023

Yes that’s what the *.aggran showed as the reason for the multiple versions. MPC excludes the latest version. Goodness knows why the narrow bounds. Goodness knows why so many just ignore the information they receive.

@jonahgraham
Copy link
Contributor Author

@merks :-( there is an error in apache's http 5 components. The import packages have too wide a range. http 5 components should allow 5.1 and 5.2 to coexist as they have all the uses clauses needed to make sure that it is wired up properly. But its imports are problematic as it allows imports to mix and match versions.

Because one part of httpcomponents imports other parts with lines like (simplified):

Bundle-Version: 5.1.3.v20221013-1742
Import-Package: org.apache.hc.core5.http;version="[5.1.3,6)"

But that means p2 can use org.apache.httpcomponents.core5.httpcore5 v5.2.3 to resolve the import. That can lead to use case problems and leads to strange resolutions by p2 where, for example, I have this set of plug-ins in one of my EPPs (CPP on Windows):

  1. org.apache.httpcomponents.client5.httpclient5_5.1.3.v20221013-1742.jar
  2. org.apache.httpcomponents.core5.httpcore5-h2_5.2.3.v20230922-1600.jar
  3. org.apache.httpcomponents.client5.httpclient5-win_5.1.3.v20221013-1742.jar
  4. org.apache.httpcomponents.httpclient_4.5.14.jar
  5. org.apache.httpcomponents.core5.httpcore5_5.1.4.v20221013-1742.jar
  6. org.apache.httpcomponents.httpcore_4.4.16.v20221207-1049.jar
  7. org.apache.httpcomponents.core5.httpcore5_5.2.3.v20230922-1600.jar

So EPP for CPP on Windows with M3 simrel (and #82 reverted) MPC still doesn't work:


!ENTRY org.eclipse.epp.mpc.core 4 0 2023-11-16 18:55:20.933
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.epp.mpc.core [224]
  Unresolved requirement: Import-Package: org.apache.hc.core5.http; version="[5.1.0,5.2.0)"

But I can see p2 coming to different solutions

@merks
Copy link
Contributor

merks commented Nov 17, 2023

FYI, I have generated the catalogs for m3 to add the -D option and to add the fixed MPC update site...

@jonahgraham
Copy link
Contributor Author

You also need to add a dependency on the httpclient ECF provider otherwise the only provider will be excluded IIUC.

@merks
Copy link
Contributor

merks commented Nov 17, 2023

Good point! That's added too now.

@jonahgraham jonahgraham mentioned this issue Nov 21, 2023
Merged
@jonahgraham
Copy link
Contributor Author

I have tested the current staging repo with the CPP package and this change looks good. I don't have a way to test with the problematic proxies, but I can see that with this change a non-existent host URL throws and exception in Apache's client now:

With this change you can see org.eclipse.ecf.provider.filetransfer.httpclient5

ENTRY org.eclipse.equinox.p2.transport.ecf 2 0 2023-11-22 12:20:13.252
!MESSAGE Connection to https://asdfasdfasfsdfdsfwerfwefwefwef.com/p2.index failed on asdfasdfasfsdfdsfwerfwefwefwef.com: Name or service not known. Retry attempt 0 started
!STACK 0
java.net.UnknownHostException: asdfasdfasfsdfdsfwerfwefwefwef.com: Name or service not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
	at org.apache.hc.client5.http.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:43)
	at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172)
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.eclipse.ecf.provider.filetransfer.httpclient5.HttpClientRetrieveFileTransfer.performConnect(HttpClientRetrieveFileTransfer.java:1009)
	at org.eclipse.ecf.provider.filetransfer.httpclient5.HttpClientRetrieveFileTransfer$1.performFileTransfer(HttpClientRetrieveFileTransfer.java:997)
	at org.eclipse.ecf.filetransfer.FileTransferJob.run(FileTransferJob.java:76)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

With the -D removed from eclipse.ini you can see org.eclipse.ecf.provider.filetransfer.httpclientjava

!ENTRY org.eclipse.equinox.p2.core 4 0 2023-11-22 12:21:11.478
!MESSAGE Provisioning exception
!STACK 1
org.eclipse.equinox.p2.core.ProvisionException: Unable to read repository at https://asdfasdfasfsdfdsfwerfwefwefwef.com/site.xml.
	at org.eclipse.equinox.internal.p2.updatesite.UpdateSite.loadActualSiteFile(UpdateSite.java:255)
	at org.eclipse.equinox.internal.p2.updatesite.UpdateSite.load(UpdateSite.java:155)
	at org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepositoryFactory.initializeRepository(UpdateSiteMetadataRepositoryFactory.java:108)
	at org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepositoryFactory.load(UpdateSiteMetadataRepositoryFactory.java:62)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:63)
	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:791)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.eclipse.oomph.util.ReflectUtil.invokeMethod(ReflectUtil.java:119)
	at org.eclipse.oomph.p2.internal.core.CachingRepositoryManager.loadRepository(CachingRepositoryManager.java:448)
	at org.eclipse.oomph.p2.internal.core.CachingRepositoryManager.loadRepository(CachingRepositoryManager.java:231)
	at org.eclipse.oomph.p2.internal.core.CachingRepositoryManager$Metadata.loadRepository(CachingRepositoryManager.java:520)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:110)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:105)
	at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.doLoad(LoadMetadataRepositoryJob.java:126)
	at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.runModal(LoadMetadataRepositoryJob.java:110)
	at org.eclipse.equinox.internal.p2.ui.sdk.PreloadingRepositoryHandler$1.runModal(PreloadingRepositoryHandler.java:84)
	at org.eclipse.equinox.p2.operations.ProvisioningJob.run(ProvisioningJob.java:188)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
	at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientRetrieveFileTransfer.performConnect(HttpClientRetrieveFileTransfer.java:1012)
	at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientRetrieveFileTransfer$1.performFileTransfer(HttpClientRetrieveFileTransfer.java:999)
	at org.eclipse.ecf.filetransfer.FileTransferJob.run(FileTransferJob.java:76)

jonahgraham added a commit to jonahgraham/packages that referenced this issue Nov 22, 2023
@jonahgraham
Add entry for ECF P2 backend change
29b8b8a 
See eclipse-packaging#81
@jonahgraham jonahgraham mentioned this issue Nov 22, 2023
Merged
@merks
Copy link
Contributor

merks commented Nov 22, 2023

Both the platform sdk and the installer have been tested with this system property and confirmed to work in the proxy environment where the new provider was failing. So I think we are good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@merks @jonahgraham