Invalid entry point for Ubuntu 20.04

[nsmlzl]

The image of Ubuntu 20.04 does not work for me. Ubuntu 22.04 works fine.

$ toolbox create --image quay.io/toolbx-images/ubuntu-toolbox:20.04
Image required to create toolbox container.
Download quay.io/toolbx-images/ubuntu-toolbox:20.04 (500MB)? [y/N]: y
Created container: ubuntu-toolbox-20.04
Enter with: toolbox enter ubuntu-toolbox-20.04

$ toolbox --verbose enter ubuntu-toolbox-20.04 
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user niklas 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /home/niklas/.config/toolbox 
DEBU Current Podman version is 3.4.4              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.4.4                  
DEBU Migration not needed: Podman version 3.4.4 is unchanged 
DEBU Resolving container and image names          
DEBU Container: 'ubuntu-toolbox-20.04'            
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'ubuntu-toolbox-20.04'            
DEBU Image: 'fedora-toolbox:33'                   
DEBU Release: '33'                                
DEBU Checking if container ubuntu-toolbox-20.04 exists 
DEBU Inspecting mounts of container ubuntu-toolbox-20.04 
DEBU Starting container ubuntu-toolbox-20.04      
DEBU Inspecting entry point of container ubuntu-toolbox-20.04 
DEBU Entry point PID is a float64                 
DEBU Entry point of container ubuntu-toolbox-20.04 is toolbox (PID=0) 
Error: invalid entry point PID of container ubuntu-toolbox-20.04

[travier]

I've just tried that and it worked for me on Fedora 36. What's your host toolbox version and host distribution?

[nsmlzl]

I am running Ubuntu 22.04 and installed toolbox v0.0.99.2 from the Ubuntu package registry.

[travier]

Can you try in a fresh virtual machine to rule out anything in your local setup?

[nsmlzl]

I was able to reproduce this in a vm (freshly created with ubuntu-22.04.1 iso). I updated all packages before installing / running toolbox.

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install podman-toolbox
$ toolbox create --image quay.io/toolbx-images/ubuntu-toolbox:20.04
Image required to create toolbox container.
Download quay.io/toolbx-images/ubuntu-toolbox:20.04 (500MB)? [y/N]: y
Created container: ubuntu-toolbox-20.04
Enter with: toolbox enter ubuntu-toolbox-20.04
$ toolbox --verbose enter ubuntu-toolbox-20.04
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user user 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /home/user/.config/toolbox 
DEBU Current Podman version is 3.4.4              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.4.4                  
DEBU Migration not needed: Podman version 3.4.4 is unchanged 
DEBU Resolving container and image names          
DEBU Container: 'ubuntu-toolbox-20.04'            
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'ubuntu-toolbox-20.04'            
DEBU Image: 'fedora-toolbox:33'                   
DEBU Release: '33'                                
DEBU Checking if container ubuntu-toolbox-20.04 exists 
DEBU Inspecting mounts of container ubuntu-toolbox-20.04 
DEBU Starting container ubuntu-toolbox-20.04      
DEBU Inspecting entry point of container ubuntu-toolbox-20.04 
DEBU Entry point PID is a float64                 
DEBU Entry point of container ubuntu-toolbox-20.04 is toolbox (PID=0) 
Error: invalid entry point PID of container ubuntu-toolbox-20.04

[travier]

Can you give us the output of podman start --attach --interactive ubnutu-toolbox-20.04 ?

[travier]

There are lost of hits for that error in the toolbx repo: https://github.com/containers/toolbox/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+Error%3A+invalid+entry+point+PID+of+container

• Container doesn't start because user.Current() fails inside the entry point containers/toolbox#1001
• "invalid entry point PID of container" when /var/lib/flatpak is mounted nosuid and/or nodev containers/toolbox#1084

[nsmlzl]

I ran this on the vm:

$ podman start --attach --interactive ubuntu-toolbox-20.04
toolbox: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by toolbox)

[travier]

This is likely containers/toolbox#871 and should be fixed by containers/toolbox#897 which is in 0.99.3. Please ask your distribution to ship that.

[jaques-sam]

It seems it's not solved on 20.04 and also not on 22.04 :-(

[travier]

Please open a new issue with more details

[debarshiray]

Duplicate of containers/toolbox#821

[debarshiray]

@andrewshadura @Jmennius could these two commits be added to the Ubuntu packages:

• containers/toolbox@112f281

• containers/toolbox@6063eb2

[andrewshadura]

I will upload a new toolbox version to Debian unstable and will try to backport those patches to stable too, but I don’t have direct upload access to Ubuntu, so that will require a lot more of effort.

[andrewshadura]

Oh, in fact, Debian’s not affected.

[andrewshadura]

Prepared an update for Jammy: https://salsa.debian.org/debian/podman-toolbox/-/commits/ubuntu/jammy
It FTBFS in GitLab CI as it builds against Debian unstable. It built fine in actual jammy.

[andrewshadura]

@debarshiray, as @panlinux points out in https://bugs.launchpad.net/ubuntu/+source/golang-github-containers-toolbox/+bug/1993888/comments/5, these two commits do not appear to be enough. Could you please double check if there’s anything else that might be missing?

[debarshiray]

Thanks for working on https://bugs.launchpad.net/ubuntu/+source/golang-github-containers-toolbox/+bug/1993888 @andrewshadura ! I was away on vacation from the 14th of July until today, and I will be away for GUADEC starting from tomorrow till the 31st of July.

Do you know the exact error? Otherwise I will debug it myself using a bunch of Ubuntu virtual machines once I am back.

The root problem is described in containers/toolbox@6063eb2 , and if you follow the references you will find containers/toolbox@6ad9c63 as a previous attempt at fixing the same problem. I also wrote a blog post describing the issue.

In short, we are using the same toolbox(1) binary from the host inside the container. This can run into ABI compatibility issues if the host's toolbox(1) was built against a newer runtime than what's available inside the container, because the runtimes are only backwards, not forward, compatible. Very often it's caused by a newer glibc on the host compared to the container. So, we ensure that the host's runtime is also present inside the container and force the toolbox(1) binary to always use it.

This is done through /run/host. On the host, it's a relative symbolic link to ../ created by systemd-tmpfiles(8) and inside the container it's a bind mount created through podman create --volume /:/run/host:rslave ....

That's the overall idea.