Docker plugin stopped starting Docker containers #1087
Comments
|
I can't duplicate this issue. The agents start reliably for me. If more details become available that make the problem consistently visible, please share that information. |
|
I face the exact same issue. Sometimes, it works, sometimes doesn't. EnvironmentI don't know if this can be one of the reasons, but the Cloud statistics shows that there are 3 agents in operation, but there are none. No Windows agents in Nodes and no docker container created in Windows. Restarting Jenkins doesn't takes any effect over it.
Is it possible that some caching is causing this? Let me know if you need any logs.... what is the log class that I have to put in the log recorder? |
|
Same here. Out of blue, controller stops spinning up new docker agent containers. Test connection for cloud node looks good, no significant errors in logs and yet no containers are being spun up. |
|
We had the issue again today. In our Jenkins, the log recorder 'com.nirima.jenkins.plugins.docker' with the log level INFO is configured. However, when the issue pops up and the plugin is not triggering jobs using labels defined in the Docker clouds, the log is like 'frozen'. Is there any way to extend/reconfigure the log(s) to get more details about what's going on while the issue exists? |
|
There have been multiple reports that the most recent release of Apache httpclient 5 API plugin breaks the docker plugin when it is trying to use a Unix domain socket to connect to the docker daemon. Refer to:
|
|
hey @MarkEWaite. I saw that apache-httpcomponents-client-5-api-plugin has a new release. Any plans to release a new docker-plugin version (maybe 1.7.1) that depends on it? Do you think it could solve #1103 ? |
A new release of the docker plugin is not needed. Users can install that new release of the httpclient 5 plugin and it will be used by the docker plugin. Even though no release is needed, I did update the docker plugin to rely on the most recent httpclient 5 plugin release in this pull request:
Yes, I've confirmed that it fixes the issue. That release fixes the issue by reverting the library inside the httpclient 5 plugin to 5.3.1. The apache-httpcomponents-client-5-api-plugin 5.4-124.v31e2987e48f4 changelog says:
I've left issue #1103 open because the permanent fix has been submitted to the docker-java library and needs to be merged and released by one of the maintainers of the docker-java library. |
|
Hey @MarkEWaite, After doing what was proposed as a fix for this issue, I realized that it didn't solve it properly. Instead, sounds like things got worse. Before, it could sometimes start more containers than now. I keep getting these messages in Cloud: java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source)
at java.base/java.net.Socket$SocketInputStream.read(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:250)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:56)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:331)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:193)
at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.HijackingHttpRequestExecutor.execute(HijackingHttpRequestExecutor.java:50)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:236)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:717)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:206)
Caused: java.lang.RuntimeException
at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:210)
at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
at PluginClassLoader for docker-java-api//com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
at PluginClassLoader for docker-java-api//com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:102)
at PluginClassLoader for docker-java-api//com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:31)
at PluginClassLoader for docker-java-api//com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:13)
at PluginClassLoader for docker-java-api//com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
at PluginClassLoader for docker-java-api//com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
at PluginClassLoader for docker-java-api//com.github.dockerjava.core.command.StartContainerCmdImpl.exec(StartContainerCmdImpl.java:42)
at PluginClassLoader for docker-plugin//com.nirima.jenkins.plugins.docker.DockerTemplate.doProvisionNode(DockerTemplate.java:759)
at PluginClassLoader for docker-plugin//com.nirima.jenkins.plugins.docker.DockerTemplate.provisionNode(DockerTemplate.java:690)
at PluginClassLoader for docker-plugin//com.nirima.jenkins.plugins.docker.DockerCloud$1.run(DockerCloud.java:414)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source) |
|
Forgot to mention that I still face the node counting issue mentioned here. |
|
Any assessment when a bug-fixed plugin version will be available? Or, would you recommend to go back to its previous version? |
@galindro and @andreasgk16 the report at the top of the issue is something that I've been unable to duplicate. I don't know of any fix for the Intermittent failure to create an agent because I can't duplicate the issue. The change in the httpclient 5 API plugin that reverted to httpclient 5.3.1 resolved the issue that I can duplicate, where agents never started when using a local connection to the docker daemon. If you can provide enough details so that others can duplicate the issue, it will increase the chances of a fix. |
Jenkins and plugins versions report
Environment
What Operating System are you using (both controller, and any agents involved in the problem)?
Jenkins master: Oracle Linux 9
Jenkins agents: Oracle Linux 9 + Windows Server 2019
Reproduction steps
The issue cannot be reproduced. In the last couple of months, we had it twice.
Symptoms:
Expected Results
The Docker plugin should react and
Actual Results
No reaction from the plugin. Triggered jobs wait endlessly for their execution as their labels cannot be resolved.
Anything else?
Current workaround: Restarting the complete Jenkins application solves the problem.
Are you interested in contributing a fix?
No response
The text was updated successfully, but these errors were encountered: