StrictHostKeyChecking=no is insecure

[elitak]

The hardcoded -o StrictHostKeyChecking=no everywhere is a big SecOps no-no. It's quite feasible an attacker could wind up with an IP address you neglect to change after relinquishing, and have an entire host config hand-delivered to him to inspect for vulnerabilities. He wouldn't be able to MITM the the deployment, but obtaining what is essentially a dump of the host's whole filesystem is still pretty disastrous, from a defensive standpoint.

Ideallly, as @oconnorr suggested in #262, nixops would have its own known_hosts mapping cached, but short of that, I'd like to at least have the option of turning on StrictHostKeyChecking=yes permanently somehow. It really should be at least =ask by default, too, but I expect handling the user interaction could cause some grief. I'd much rather have to ssh-keygen -R $host; ssh $host : every time a host changes its keys, than compromise a big part of ssh's integrity for a small convenience.

[domenkozar]

For context:

     StrictHostKeyChecking
	     If	this flag is set to ``yes'', ssh(1) will never automatically
	     add host keys to the ~/.ssh/known_hosts file, and refuses to con-
	     nect to hosts whose host key has changed.	This provides maximum
	     protection	against	trojan horse attacks, though it	can be annoy-
	     ing when the /etc/ssh/ssh_known_hosts file	is poorly maintained
	     or	when connections to new	hosts are frequently made.  This
	     option forces the user to manually	add all	new hosts.  If this
	     flag is set to ``no'', ssh	will automatically add new host	keys
	     to	the user known hosts files.  If	this flag is set to ``ask'',
	     new host keys will	be added to the	user known host	files only
	     after the user has	confirmed that is what they really want	to do,
	     and ssh will refuse to connect to hosts whose host	key has
	     changed.  The host	keys of	known hosts will be verified automati-
	     cally in all cases.  The argument must be ``yes'',	``no'',	or
	     ``ask''.  The default is ``ask''.

[elitak]

Incidentally, the wording of that paragraph is quite misleading, especially: The host keys of known hosts will be verified automatically in all cases.. That sounds to me like ssh will refuse to connect on a mismatch with the setting =no, but it does anyway. Only password auth and agent forwarding are forcefully disabled when there's a mismatch.

Also, semantically, what is a "strict check" anyway? The check is already done every time; strictness applies to policy enforcement. The option conflates two distinct concerns and should be divided into respective options: something like AutoAddKnownHosts=always|if-new|never|ask and KnownHostKeyMatchPolicy=strict|no-mismatch|lax|ask . Perhaps I'll file this upstream.

[nh2]

Can we narrow down the "everywhere" a bit so that it's clear what parts are problematic?

$ git grep StrictHostKey
nix/ssh-tunnel.nix:       + " -o StrictHostKeyChecking=no"
nixops/backends/digital_ocean.py:            '-o', 'StrictHostKeyChecking=no',
nixops/backends/hetzner.py:             "-o", "StrictHostKeyChecking=no"]
nixops/backends/hetzner.py:            ["-o", "StrictHostKeyChecking=no",
nixops/backends/libvirtd.py:        return super_flags + ["-o", "StrictHostKeyChecking=no",
nixops/backends/none.py:            return super_state_flags + ["-o", "StrictHostKeyChecking=no", "-i", self.get_ssh_private_key_file()]
tests/none-backend.nix:      $coordinator->succeed("ssh -o StrictHostKeyChecking=no -v target1 ls / >&2");
tests/none-backend.nix:      $coordinator->succeed("ssh -o StrictHostKeyChecking=no -v target2 ls / >&2");
tests/none-backend.nix:                           "    -o StrictHostKeyChecking=no target1 : >&2");

It seems to me that for libvirtd, the flag is OK. AWS, GCE and Azure aren't present here, probably because they already have a secure host key fetching mechanism and thus don't need StrictHostKeyChecking=no.

So remaining are these 4 problematic cases:

• ssh-tunnel
• digital_ocean
• hetzner
• none

[elitak]

I reckon that nixops should use ssh-keyscan to obtain and store the host keys in the database (when the host is first deployed), thereafter generating in tmpfs a known_hosts file, whenever ssh needs to be run. Ssh commands that formerly used -o StrictHostKeyChecking=no instead use GlobalKnownHostsFile=/tmp/known_hosts.XXX -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=yes. Host keys are deleted from the db only when the machine is deleted or a new command like nixops clear-hostkey <machine> is run. Another command like nixops set-hostkey <machine> "<line from ssh-keyscan>" would be useful as well, for the paranoid.

All this is rather clunky, but I don't see a better way of doing it correctly, short of using some protocol-level ssh lib like paramiko to avoid the need to have the hostkeys on disk. I'm not too familiar with the nixops code, so I'm reluctant to attempt these changes without more input.

[sorki]

I've dropped this in vpsfreecz@c0affe0 completely to follow KISS principles and just use ~/.ssh/known_hosts as usual.

In vpsfreecz@510485d I've added a dumb backend which is like none backend but without keypair generation/storage in state - should respect ~/.ssh/config and allow for password and key pair authentication if public key is deployed by other means.

What do you think? Can revise these and create pull requests if it makes sense.

[elitak]

That suits my purposes. I'm happy to manage everything ssh-related in .ssh/known_hosts and .ssh/config, although perhaps there should be a way to append -o <ArbitraryOptions> to the ssh commandline? I can't think of any situation where it wouldn't be possible to handle all special behavior within a singular ssh_config, through the use of Match stanzas, but command-line tampering might still be more convenient, to some users.

Thanks for the effort in addressing this; I'd long forgotten about it.

[aszlig]

Probably it would be a better idea to add functionality for managing ssh host keys in ssh_util, so that they're saved in the state db on first use. Back then when I wrote this comment I was deploying around 90 servers and it's really tedious to accept the host key for every single one.

However, I'd still set StrictHostKeyChecking to no for the Hetzner rescue system until we have that functionality, because on every reboot into rescue, you'll get a new host key.

Regarding the dumb backend: I'd just add an option for the none backend to not store the key in the state db.

[elitak]

I would use ssh-keyscan in cases like that, to deliberately reconstruct the known_hosts file. Keeping separate the intentions of performing a remote operation and changing host signatures is important, I think.

[coretemp]

The correct solution is to set the hostkey in Nix configuration. Preferably automatically with manual overrides depending on the backend. Any mechanism which depends on StrictHostKeyChecking not equal to yes is flawed.

I find it scary that anyone ever thought it was a good idea to implement it in the way it is currently working. It's basically a root kit on an infrastructure level for sufficiently motivated adversaries.

If this would be more popular software, it would receive a CVE number.

[aszlig]

Okay, I did actually misunderstand the implications of setting this to no (as written in the edit of https://discourse.nixos.org/t/green-carbon-neutral-nixos-hosting/1187/10), so while #1023 is not a full fix for this issue, setting it to accept-new would at least mitigate the most scary parts about it.

[aszlig]

Okay, Hetzner does have an API enpoint to query the host key of the rescue system: https://robot.your-server.de/doc/webservice/en.html#get-boot-server-ip-rescue-last

I've added aszlig/hetzner#32 accordingly to get this into the library so we can make use of it via NixOps.

[coretemp]

Thanks @aszlig. The more backends we have, the better the cloud environment becomes.

Somewhat related to this is using nixops ssh to login to EC2 machines, which currently gives a choice, but it would be nicer if it would use EC2 APIs to also ensure automated security.

[grahamc]

We don't use =no anymore.