Vagrant halt, freeze with SSH Sending keep alive

[KartoffelToby]

System: Mac OS X
Vagrant 1.7.4
VMware-Fusion 7

vagrant hangs on vagrant halt and vagrant reload.

Here is the output from vagrant halt --debug

....
DEBUG ssh: PTY stdout parsed: Debian GNU/Linux stretch/sid \n \l

INFO guest: Detected: debian!
DEBUG guest: Searching for cap: halt
DEBUG guest: Checking in: debian
DEBUG guest: Checking in: linux
DEBUG guest: Found cap: halt in linux
INFO guest: Execute capability: halt #<Vagrant::Machine: BOXNAME (HashiCorp::VagrantVMwarefusion::Provider)>
DEBUG ssh: Re-using SSH connection.
INFO ssh: Execute: shutdown -h now (sudo=true)
DEBUG ssh: pty obtained for connection
DEBUG ssh: stdout: export TERM=vt100

DEBUG ssh: Re-using SSH connection.
INFO ssh: Execute: echo; printf $SSH_AUTH_SOCK (sudo=false)
DEBUG ssh: pty obtained for connection
DEBUG ssh: stdout: export TERM=vt100
stty raw -echo
export PS1=
export PS2=
export PROMPT_COMMAND=
printf bccbb768c119429488cfd109aacea6b5-pty
echo; printf $SSH_AUTH_SOCK
exitcode=$?
printf bccbb768c119429488cfd109aacea6b5-pty
exit $exitcode

DEBUG ssh: stdout: root@kartoffel-box:/home/vagrant# export TERM=vt100
root@kartoffel-box:/home/vagrant#
DEBUG ssh: stdout: vagrant@kartoffel-box:$ expor
DEBUG ssh: stdout: t
DEBUG ssh: stdout: TERM=vt10
DEBUG ssh: stdout: 0

DEBUG ssh: stdout: vagrant@kartoffel-box:$ stty r
DEBUG ssh: stdout: aw -
DEBUG ssh: stdout: e
DEBUG ssh: stdout: cho

DEBUG ssh: stdout: vagrant@kartoffel-box:$
DEBUG ssh: stdout: bccbb768c119429488cfd109aacea6b5-pty
DEBUG ssh: stdout:

DEBUG ssh: stdout: /tmp/ssh-NytPFZbPrZ/agent.2229
DEBUG ssh: stdout: bccbb768c119429488cfd109aacea6b5-pty
DEBUG ssh: stdout: Abgemeldet

DEBUG ssh: Exit status: 0
DEBUG ssh: PTY stdout: export TERM=vt100
stty raw -echo
export PS1=
export PS2=
export PROMPT_COMMAND=
printf bccbb768c119429488cfd109aacea6b5-pty
echo; printf $SSH_AUTH_SOCK
exitcode=$?
printf bccbb768c119429488cfd109aacea6b5-pty
exit $exitcode
vagrant@kartoffel-box:$ export TERM=vt100
vagrant@kartoffel-box:$ stty raw -echo
vagrant@kartoffel-box:$ bccbb768c119429488cfd109aacea6b5-pty
/tmp/ssh-NytPFZbPrZ/agent.2229bccbb768c119429488cfd109aacea6b5-ptyAbgemeldet

DEBUG ssh: PTY stdout parsed:
/tmp/ssh-NytPFZbPrZ/agent.2229
INFO ssh: Setting SSH_AUTH_SOCK remotely: /tmp/ssh-NytPFZbPrZ/agent.2229
nt.2229sh: stdout: export SSH_AUTH_SOCK=/tmp/ssh-NytPFZbPrZ/age
root@kartoffel-box:/home/vagrant# stty raw -echo

DEBUG ssh: stdout: root@kartoffel-box:/home/vagrant#
DEBUG ssh: stdout: bccbb768c119429488cfd109aacea6b5-pty
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
...

[sethvargo]

Duplicate of #6051. Please see my solution there. Thanks! 😄

[legal90]

@sethvargo We've run into the same issue with parallels provider: Parallels/vagrant-parallels#240
However, It is reproducible even with minimal configuration ( vagrant init -m bento/debian-8.1 && vagrant up --provider=parallels), so your suggestion above could not be applied.

Like in the top message, communicator is sending SSH keep-alive, so there is nothing to fix on the provider side. Should this bug be reopened or we have to file the new one?

P.S. I could reproduce it only with Debian guest. Other guests (CentOS, Ubuntu) are halting fine.
P.P.S. I couldn't reproduce it with virtualbox provider (even on the same version of box bento/debian-8.1). I'm not sure, but probably it is caused by the fact that virtualbox operates SSH connections via forwarded ports, but parallels uses direct IP connection.

[dch]

Just reporting I have this too using debian sid (latest 4.3 kernel). I'm happy to provide more logs or help further.

sample vagrant parallels box is https://atlas.hashicorp.com/ideegeo/boxes/debian-sid

 INFO subprocess: Starting process: ["/usr/local/bin/prlctl", "list", "91e45743-0560-45c0-9143-077ecf538404", "--no-header", "-o", "status"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stdout: running
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 31999
DEBUG subprocess: Exit status: 0
 INFO interface: output: Attempting graceful shutdown of VM...
 INFO interface: output: ==> default: Attempting graceful shutdown of VM...
==> default: Attempting graceful shutdown of VM...
DEBUG ssh: Checking whether SSH is ready...
DEBUG ssh: Re-using SSH connection.
 INFO ssh: SSH is ready!
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute:  (sudo=false)
DEBUG ssh: Exit status: 0
DEBUG guest: Searching for cap: halt
DEBUG guest: Checking in: debian
DEBUG guest: Checking in: linux
DEBUG guest: Found cap: halt in linux
 INFO guest: Execute capability: halt [#<Vagrant::Machine: default (VagrantPlugins::Parallels::Provider)>] (debian)
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute: shutdown -h now (sudo=true)
DEBUG ssh: stderr: mesg: ttyname failed: Inappropriate ioctl for device

DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...

[cliffrowley]

Is anyone currently looking at this? It's driving me mad and I'm stuck using Parallels currently because I need its nested virtualisation. If no-one is looking at it I'll likely take a look myself but I'm not familiar with Vagrant's source.

[cliffrowley]

I'll assume that's a no, then.

[Sharpie]

This has also been reported in oscar-stack/vagrant-hosts#65

One thing I noticed is that some of these DEBUG traces show a PTY being allocated after a command is executed on the SSH session. Is this correct behavior? Shouldn't the PTY be allocated first?

[DanHam]

Hi,

This issue is probably specific to Debian/Debian derivatives and limited to boxes running systemd. There is a very long running thread here where the issue is discussed in some detail! Read through for the full gory details.

Long and short, I managed to prevent this behaviour with my Debian box by:

1/ Ensuring the libpam-systemd package is installed in the guest

sudo apt-get install libpam-systemd

2/ Ensuring that /etc/ssh/sshd_config has the required option set to use PAM

UsePAM yes

3/ Restarting the box - you'll have to put up with the issue one last time... CTRL-C to the rescue!

Once this is taken care of issuing a vagrant halt behaves as expected and everything shuts down cleanly. Similarly, when issuing a 'shutdown -h now' from within an ssh session, the ssh session is automatically closed and control returned to the shell.

Note that the libpam-systemd package is 'recommended' for installation when systemd is installed. Debian systems should install 'recommended' packages by default so any box exhibiting this behaviour has probably been built with the APT setting of Install-Recommends set to false (see /etc/apt/apt.conf.d/00InstallRecommends).

Just for completeness:

Host OS: OS X 10.10.5
Vagrant version: 1.8.5
Virtualisation: VMware Fusion Professional Version 8.1.1 (3771013)
Debian VM: 8.5 (Jessie)

Hope that helps.

Dan

[chrisroberts]

@DanHam 's comment sums this up pretty nicely: #6207 (comment)

I'm going to close this up. If the behavior is still seen with properly configured guests, please open a new issue. Thanks!

[vdloo]

If @DanHam's solution above does not work, try creating a oneshot systemd service that kills sshd like explained here

# cat >/etc/systemd/system/kill-ssh-user-sessions-on-shutdown.service <<EOF
[Unit]
Description=Shutdown all ssh sessions before network
DefaultDependencies=no
Before=network.target shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/killall sshd

[Install]
WantedBy=poweroff.target halt.target reboot.target
EOF

# systemctl enable kill-ssh-user-sessions-on-shutdown

[flaugher]

I can confirm that vdloo's solution worked for me. I'm running VirtualBox 5.2.12, Vagrant 2.1.2, and Debian 8.11.

[DanHam]

@vdloo @flaugher Using a systemd unit to kill ssh should not be necessary if the VM you are using has the libpam-systemd package installed. That workaround is actually in the same bug report that the eventual fix was reported in!

Hope that helps.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.