bootBuildImage task hangs while saving docker image intermittently

[xuanswe]

1. Download source code from start.spring.io with the configuration below
   
2. Run ./gradlew :clean :bootBuildImage
3. If successful, update version in build.gradle.kts and repeat from step 2 above until docker is frozen. I can reproduce the bug after 4 or 5 tries.

Happens on both Spring Boot 3 and Spring Boot 2

console-log-spring-boot-3-success.txt

console-log-spring-boot-3-frozen.txt

[scottfrederick]

@xuan-nguyen-swe I can run a similar configuration successfully, so I don't think this has anything to do with the buildpack.

Does this also happen when you don't configure custom buildpacks, or only when you list the buildpacks shown above?

Can you try this using the pack CLI and see what happens? That would help determine if this has anything to do with Spring Boot or if it is related to your local machine. The equivalent pack command when run from the root of your project would be something like this:

pack build -e BP_JVM_VERSION=17 --buildpack "paketo-buildpacks/ca-certificates" --buildpack "gcr.io/paketo-buildpacks/azul-zulu" --buildpack "paketo-buildpacks/java" -p [path to built jar file] test-image

[xuanswe]

@scottfrederick Thanks for your prompt reply.
I tried again with a clean project, and it works. So, it must be smt else incompatible in my original project.
But it's strange. It works if I don't customize configuration of buildpacks.
Let's me check again and give you more information if I can find smt.

[dmikusa]

Could it be a network issue? When you don't set any buildpacks, it'll load those from the builder which is just a single image that's needed.

When you put in the buildpacks, more network requests happen. It needs to go fetch those images. Maybe one of those requests is hanging? Maybe try docker pull those specific images.

[xuanswe]

Could it be a network issue?

No, I don't think so. It hangs at the last step to save the built image as local docker image.
I am trying to reduce my project until I can reproduce with the simplest gradle project then I will post it again here.

[xuanswe]

@scottfrederick @dmikusa I updated the original post with the easiest way to reproduce the bug.

[fabioportieri]

i have the exact same issue, just opened a ticket: spring-projects/spring-boot#34026
i managed to reproduce it with a brand new application, it happens in distinct windows machine (win10 and win11)
i'll see if i can reproduce it on my linux machine too

[dmikusa]

@fabioportieri what are you using for Docker (software/version)? When it's saving and your Docker becomes unresponsive, can you do other tasks with Docker? Like docker ps?

[scottfrederick]

@xuan-nguyen-swe and @fabioportieri Can you try to reproduce the problem using the pack CLI as requested earlier? This would be very helpful for us to understand if the problem has anything to do with the Spring Boot plugins.

[xuanswe]

@xuan-nguyen-swe and @fabioportieri Can you try to reproduce the problem using the pack CLI as requested earlier? This would be very helpful for us to understand if the problem has anything to do with the Spring Boot plugins.

I will try to use cli and tell me result later. I am totally new to buildpacks so I need to check how to set it up.

[fabioportieri]

docker becomes completely unresponsive even from commandline, the only thing i can do is to restart the machine, the issue has been reproduced in two different windows machines (win10 and win11), as i said i will try to reproduce in a native ubuntu installation,
regarding the pack cli i tried and i had the same issue

in my case i had to do the following commands:

pack config default-builder paketobuildpacks/builder:base 

pack build -e BP_JVM_VERSION=11 --buildpack "paketo-buildpacks/ca-certificates" --buildpack "paketo-buildpacks/bellsoft-liberica" --buildpack "paketo-buildpacks/syft" --buildpack "paketo-buildpacks/executable-jar" --buildpack "paketo-buildpacks/dist-zip" --buildpack "paketo-buildpacks/spring-boot" -p C:\lavoro\workspace\fdp-batch\dataflowmanager\target\dataflowmanager-1.0-SNAPSHOT.jar localhost:5000/dataflow-manager:1.0-SNAPSHOT

eventually stuck here, with docker unresponsive:

  Launch Helper: Reusing cached layer
  Spring Cloud Bindings 1.11.0: Reusing cached layer
  Web Application Type: Reusing cached layer
  4 application slices
  Image labels:
    org.opencontainers.image.title
    org.opencontainers.image.version
    org.springframework.boot.spring-configuration-metadata.json
    org.springframework.boot.version
    org.springframework.cloud.dataflow.spring-configuration-metadata.json
===> EXPORTING
Reusing layer 'paketo-buildpacks/ca-certificates:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:jre'
Reusing layer 'paketo-buildpacks/executable-jar:classpath'
Reusing layer 'paketo-buildpacks/spring-boot:helper'
Reusing layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Reusing layer 'paketo-buildpacks/spring-boot:web-application-type'
Reusing layer 'launch.sbom'
Reusing 5/5 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.spring-configuration-metadata.json'
Adding label 'org.springframework.boot.version'
Adding label 'org.springframework.cloud.dataflow.spring-configuration-metadata.json'
Setting default process type 'web'
Saving localhost:5000/dataflow-manager:1.0-SNAPSHOT...

[scottfrederick]

regarding the pack cli i tried and i had the same issue

Thanks for trying that @fabioportieri. That confirms that the problem is not related to the Spring Boot plugins or the pack CLI directly, and allows us to focus the troubleshooting at the layers shared by both tools.

[xuanswe]

Thanks @fabioportieri for testing the issue with pack cli.
I think it's quite clear now, so I don't need to test pack cli again.

[dmikusa]

@fabioportieri what are you using for Docker (software/version)? I assume Docker Desktop for Windows, but if you could confirm & let us know the specific version as well. If it's not the latest version, please update. Thanks

[fabioportieri]

yes, docker desktop, updated to latest version v4.16.3, for windows

this is my docker info

Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.10.0)
compose: Docker Compose (Docker Inc., v2.15.1)
dev: Docker Dev Environments (Docker Inc., v0.0.5)
extension: Manages Docker extensions (Docker Inc., v0.2.17)
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
scan: Docker Scan (Docker Inc., v0.23.0)

Server:
Containers: 37
Running: 13
Paused: 0
Stopped: 24
Images: 14
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.15.79.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 15.47GiB
Name: docker-desktop
ID: 32I6:KQAG:IIZJ:G5OH:ASNH:JFHJ:4RAS:KXEY:LG2C:HDZN:7VWJ:AFZO
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false

[anthonydahanne]

Hello,
I've tried to reproduce the issue in a public CI, for everybody to see how to reproduce.

I've successfully built on a Windows 10 machine,

Nom du système d’exploitation:              Microsoft Windows 10 Professionnel
Version du système:                         10.0.19044 N/A version 19044
Type du système:                            x64-based PC
Processeur(s):                              1 processeur(s) installé(s).
                                            [01] : Intel64 Family 6 Model 142 Stepping 9 GenuineIntel ~2703 MHz

with that ~/.wslconfig

[wsl2]
#kernel=C:\\temp\\myCustomKernel
memory=4GB # Limits VM memory in WSL 2 to 4 GB
processors=2 # Makes the WSL 2 VM use two virtual processors

and Docker For Windows v20.10.8 with Enable integration with my default WSL distro

• a JVM based image from a simple Spring Boot project using Pack CLI
• a JVM based image from a simple Spring Boot project using Spring Boot Maven Plugin
• a Native based image, from the same Spring boot project using Spring Boot Maven Plugin

a fair number of times now: https://github.com/anthonydahanne/spring-boot-demo/actions/workflows/windows.yml

@fabioportieri @xuan-nguyen-swe : I invite you to clone my repo and use your own machine as a runner, see if you still can reproduce your issues: the advantage of using GH actions is that it will allow us to investigate further and understand exactly under which conditions we can reproduce the issue.

Without any additional context / logs / recorded builds, I'd say the issue does not happen anymore.

Thanks!

[anthonydahanne]

can we close the issue?

[Azbesciak]

I also confirm the same issue, I am using spring boot, mostly with AOT builds but I also reproduced that with non-aot build.
I have that since January, 70% of builds crashed docker, the only solution is to restart the computer (I even cannot kill docker); comp works normally, but I will not run any docker image, and that one is hanging.

[anthonydahanne]

Can you please share a project that makes docker hang, along with the command you used?
In a previous comment, I stated I built dozen of times a project on Windows, without any issues.
So every detail helps here.
Thanks