Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues encountered at various steps #1

Open
airmetal opened this issue Nov 21, 2016 · 47 comments
Open

Issues encountered at various steps #1

airmetal opened this issue Nov 21, 2016 · 47 comments

Comments

@airmetal
Copy link

When I run setup.py after configuring terraform/main.tf I get the following error -

Detecting client IP...
Traceback (most recent call last):
File "./setup.py", line 84, in
detect_client_ip(local_vars)
File "./setup.py", line 62, in detect_client_ip
socket.gethostbyname('ifconfig.co') # We need the IPv4 address

I changed the setup.py to derive the URL differently. It seems its making the call now, but now I get the following error:

Detecting client IP...
Traceback (most recent call last):
File "./setup.py", line 96, in
detect_client_ip(local_vars)
File "./setup.py", line 71, in detect_client_ip
urllib2.urlopen(request).read()
File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>

The firewall is open. It seems I am missing some step. I would imagine a web server would be running prior to receiving the HTTP request?

@tintoy
Copy link
Contributor

tintoy commented Nov 21, 2016

Actually you could just use requests.Get(‘https://ipv4.ifconfig.co/json')

On 22 Nov 2016, at 5:07 AM, wninobla <[email protected]mailto:[email protected]> wrote:

When I run setup.py after configuring terraform/main.tf I get the following error -

Detecting client IP...
Traceback (most recent call last):
File "./setup.py", line 84, in
detect_client_ip(local_vars)
File "./setup.py", line 62, in detect_client_ip
socket.gethostbyname('ifconfig.cohttp://ifconfig.co') # We need the IPv4 address

I changed the setup.py to derive the URL differently. It seems its making the call now, but now I get the following error:

Detecting client IP...
Traceback (most recent call last):
File "./setup.py", line 96, in
detect_client_ip(local_vars)
File "./setup.py", line 71, in detect_client_ip
urllib2.urlopen(request).read()
File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>

The firewall is open. It seems I am missing some step. I would imagine a web server would be running prior to receiving the HTTP request?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezKhaMYF6X5zeKBA3Ks5vpuYYTwlFks5rAd3GgaJpZM4K4hDS.

itevomcid

@airmetal
Copy link
Author

Did not work either.

@tintoy
Copy link
Contributor

tintoy commented Nov 21, 2016

Oops - wrong URL.

https://v4.ifconfig.co/json

Otherwise if it's a local network problem, you'll need to manually supply your client IP (in which case it won't try to auto-detect).

@airmetal
Copy link
Author

airmetal commented Nov 22, 2016

Ok thanks. I was having issues with python versions and SSL support. So finally got past with Python 3.5 and updated code. Although, ran into this later when running terraform plan.

module root: 3 error(s) occurred:
Variable 'client_ip': duplicate found. Variable names must be unique.
Variable 'ssh_public_key_file': duplicate found. Variable names must be unique.
Variable 'ssh_bootstrap_password': duplicate found. Variable names must be unique.

So I removed the variables from the main.tf.

Its running now :)

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Hmm - the variables should be declared in there without a default value (just { } instead of { default = "xxx" }).

On 22 Nov 2016, at 2:19 PM, Andrew Das <[email protected]mailto:[email protected]> wrote:

Ok thanks. I was having issues with python versions and SSL support. So finally got past with Python 3.5 and updated code. Although, ran into this later when running terraform plan.

module root: 3 error(s) occurred:
Variable 'client_ip': duplicate found. Variable names must be unique.
Variable 'ssh_public_key_file': duplicate found. Variable names must be unique.
Variable 'ssh_bootstrap_password': duplicate found. Variable names must be unique.

So I removed the variables from the main.tf.

Its running now :)


You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262136943, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezELJtqJrc00-NBcBwS-XuNC13oL7ks5rAl8ngaJpZM4K4hDS.

itevomcid

@airmetal
Copy link
Author

Ok VM's deployed but now seeing this:

Error applying plan:

4 error(s) occurred:

  • ddcloud_server.rancher_host: Failed to apply 1 tags to server 'b5a506e3-cad3-4d87-bc67-8620358754eb' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.
  • ddcloud_server.worker.2: Failed to apply 1 tags to server 'a79e79eb-41ea-4553-8917-c19e2a8ed329' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.
  • ddcloud_server.worker.0: Failed to apply 1 tags to server 'f7cc72ef-2819-4823-8ab9-5b8a41097359' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.
  • ddcloud_server.worker.1: Failed to apply 1 tags to server '8a31ca03-d847-4de3-b3c1-469dce97c54f' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Do you have permission to define tag keys?

On 22 Nov 2016, at 2:25 PM, Andrew Das <[email protected]mailto:[email protected]> wrote:

Ok VM's deployed but now seeing this:

Error applying plan:

4 error(s) occurred:

  • ddcloud_server.rancher_host: Failed to apply 1 tags to server 'b5a506e3-cad3-4d87-bc67-8620358754eb' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.
  • ddcloud_server.worker.2: Failed to apply 1 tags to server 'a79e79eb-41ea-4553-8917-c19e2a8ed329' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.
  • ddcloud_server.worker.0: Failed to apply 1 tags to server 'f7cc72ef-2819-4823-8ab9-5b8a41097359' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.
  • ddcloud_server.worker.1: Failed to apply 1 tags to server '8a31ca03-d847-4de3-b3c1-469dce97c54f' (response code 'RESOURCE_NOT_FOUND'): Tag Key(s) (roles) not found.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.


You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262137681, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezG3UPu2ZS-0z0bqT8C__zZBDeZXjks5rAmCzgaJpZM4K4hDS.

itevomcid

@airmetal
Copy link
Author

Yes I do.

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Ok, can you please set the following environment variables and run terraform apply again?

  • TF_LOG=DEBUG
  • MCP_EXTENDED_LOGGING=1

Let's see what the log says.

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

(it should be automatically creating the tag keys because it's been configured to)

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

BTW if you're stuck and just need to get going, you just need to go and define the tag key "roles" and everything will work.

But if you could run with logging once before you go and manually create the tag key I'd appreciate it (feel free to email me the logs instead). It'd be useful for working out where the problem lies :)

@airmetal
Copy link
Author

Sounds good. Sure np. Should I generate the logs with terraform apply?

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Yes please - just make sure those variables are set first :)

On 22 Nov 2016, at 2:53 PM, Andrew Das <[email protected]mailto:[email protected]> wrote:

Sounds good. Sure np. Should I generate the logs with terraform apply?


You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262140906, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezIff0DiN3m1ih8JLe-XG8Bx5GzBtks5rAmcngaJpZM4K4hDS.

itevomcid

@airmetal
Copy link
Author

Where is the log generated?

@airmetal
Copy link
Author

Ok log attached.

Rancher-Terraform-Log.txt

@airmetal
Copy link
Author

airmetal commented Nov 22, 2016

Ok I added the roles tag key. Then ran ansible-playbook but Ansible cannot connect to workers. BTW my client machine is an MCP VM on another Network Domain. I gave the Client IP value as the public IP of the client machine (RHEL6). In case that was an issue.

(.dd-mcp) [root@10-0-0-9 ansible]# ansible-playbook playbooks/upgrade-packages.yml
/root/.dd-mcp/lib/python2.6/site-packages/cryptography/init.py:26: DeprecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of cryptography will drop support for Python 2.6
DeprecationWarning

PLAY [all] *********************************************************************

TASK [clean apt] ***************************************************************
fatal: [worker-01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [rancher]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [worker-02]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [worker-03]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
to retry, use: --limit @/root/rancher-ddcloud/ansible/playbooks/upgrade-packages.retry

PLAY RECAP *********************************************************************
rancher : ok=0 changed=0 unreachable=1 failed=0
worker-01 : ok=0 changed=0 unreachable=1 failed=0
worker-02 : ok=0 changed=0 unreachable=1 failed=0
worker-03 : ok=0 changed=0 unreachable=1 failed=0

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Ah no, if it's in the MCP the client IP will need to be the client machine's network domain S/NAT IP address (firewall rules reference it).

TBH, I didn't envision the client machine being in CaaS when I built this. Could work but there may need to be tweaks.

On 22 Nov. 2016, at 3:18 pm, Andrew Das <[email protected]mailto:[email protected]> wrote:

Ok Ansible cannot connect to workers. BTW my client machine is an MCP VM on another Network Domain. I gave the Client IP value as the public IP of the client machine (RHEL6). In case that was in issue.

(.dd-mcp) [root@10-0-0-9 ansible]# ansible-playbook playbooks/upgrade-packages.yml
/root/.dd-mcp/lib/python2.6/site-packages/cryptography/init.py:26: DeprecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of cryptography will drop support for Python 2.6
DeprecationWarning

PLAY [all] *********************************************************************

TASK [clean apt] ***************************************************************
fatal: [worker-01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [rancher]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [worker-02]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
fatal: [worker-03]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
to retry, use: --limit @/root/rancher-ddcloud/ansible/playbooks/upgrade-packages.retry

PLAY RECAP *********************************************************************
rancher : ok=0 changed=0 unreachable=1 failed=0
worker-01 : ok=0 changed=0 unreachable=1 failed=0
worker-02 : ok=0 changed=0 unreachable=1 failed=0
worker-03 : ok=0 changed=0 unreachable=1 failed=0

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262143639, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezCxPmqHgtrYAXo4ZrPUy6K9cAsYyks5rAmzygaJpZM4K4hDS.

itevomcid

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

@DDDrew - FYI, the auto_create_tag_keys feature has been removed; the CloudControl API it uses is too fragile, and considering how infrequently it would actually be useful to have I don't consider it worth maintaining (creation of tag keys is not really something worth automating - it's done once per client organisation and can only be done in their home region).

@airmetal
Copy link
Author

Ok no problem. I can run this from my Mac. Also try out the latest code w/o tagging.

@airmetal
Copy link
Author

airmetal commented Nov 22, 2016

Ok everything works until Ansible tries to make a connection. It gets permission denied with the password. I had also commented out the variables in main.tf otherwise I was getting an error regarding duplicate variables.

Also I notice during provisioning the following in the output. Is that expected? Making sure its picking up the values.

I will stop here until I get further directions from you. Don't want to tinker around further or it may cause some confusion.

null_resource.install_ssh_key.1:
null_resource.install_ssh_key.2:
null_resource.install_ssh_key.0:
....

@airmetal airmetal changed the title Detecting Client IP address issue Issues encountered at various steps Nov 22, 2016
@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

What if you try running ssh manually?

ssh -i my/keyfile [email protected]:[email protected]

Can you connect then? Usually access denied from Ansible is due to something like a key mismatch in ~/.ssh/known_hosts or similar.

On 23 Nov. 2016, at 12:01 am, Andrew Das <[email protected]mailto:[email protected]> wrote:

Ok everything works until Ansible tries to make a connection. It gets permission denied with the password.

Also I notice during provisioning the following in the output. Is that expected? Making sure its picking up the values.

null_resource.install_ssh_key.1:
null_resource.install_ssh_key.2:
null_resource.install_ssh_key.0:
....

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262234647, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezCu7IJUwzp_n-um_o7gQRj3EpyFHks5rAuedgaJpZM4K4hDS.

itevomcid

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

BTW Ansible should not be using a password- that's why those null_resources are used - to install SSH keys (disables password authentication after the key is installed).

On 23 Nov. 2016, at 12:01 am, Andrew Das <[email protected]mailto:[email protected]> wrote:

Ok everything works until Ansible tries to make a connection. It gets permission denied with the password.

Also I notice during provisioning the following in the output. Is that expected? Making sure its picking up the values.

null_resource.install_ssh_key.1:
null_resource.install_ssh_key.2:
null_resource.install_ssh_key.0:
....

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262234647, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezCu7IJUwzp_n-um_o7gQRj3EpyFHks5rAuedgaJpZM4K4hDS.

itevomcid

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

BTW, the null_resource provisioners will only be run once. I take it the SSH key file path used by those provisioners is correct?

You could run terraform destroy -force followed by TF_LOG=DEBUG terraform apply and watch for the null_resource bits (should show you the SSH commands being run).

On 23 Nov 2016, at 12:01 AM, Andrew Das <[email protected]mailto:[email protected]> wrote:

Ok everything works until Ansible tries to make a connection. It gets permission denied with the password.

Also I notice during provisioning the following in the output. Is that expected? Making sure its picking up the values.

null_resource.install_ssh_key.1:
null_resource.install_ssh_key.2:
null_resource.install_ssh_key.0:
....


You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262234647, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezCu7IJUwzp_n-um_o7gQRj3EpyFHks5rAuedgaJpZM4K4hDS.

itevomcid

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Also, you can pass -vvvv to ansible / ansible-playbook for full verbosity.

On 23 Nov 2016, at 12:01 AM, Andrew Das <[email protected]mailto:[email protected]> wrote:

Ok everything works until Ansible tries to make a connection. It gets permission denied with the password.

Also I notice during provisioning the following in the output. Is that expected? Making sure its picking up the values.

null_resource.install_ssh_key.1:
null_resource.install_ssh_key.2:
null_resource.install_ssh_key.0:
....


You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262234647, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezCu7IJUwzp_n-um_o7gQRj3EpyFHks5rAuedgaJpZM4K4hDS.

itevomcid

@airmetal
Copy link
Author

airmetal commented Nov 22, 2016

What about the duplicate variables? The local_vars.tf also has the same variables so that's where the duplication is happening. I imagine that's expected? I am seeing this:

terraform plan
module root: 3 error(s) occurred:
Variable 'client_ip': duplicate found. Variable names must be unique.
Variable 'ssh_public_key_file': duplicate found. Variable names must be unique.
Variable 'ssh_bootstrap_password': duplicate found. Variable names must be unique.

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

From what I understand, you can declare a variable as many times as you like as long as you only supply a value for it in one place.

I've just checked out a fresh copy of the repository to test this out, and I don't see that error message. Is there any chance a file has been copied or otherwise doubled up?

Maybe do a find-in-files in the terraform directory for ssh_public_key_file and see how many results you get?

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

BTW, if you get the latest source, you'll want to delete terraform/dns.tf (since I have now merged the 2 branches I was working on; the one with DNS support is newer so it might be worth trying that one; if you delete terraform/dns.tf then setup.py knows you're not using DNS and configures things accordingly.

If you checked out the repository using Git, it might be worth running git status and posting the output here.

@tintoy tintoy self-assigned this Nov 22, 2016
@airmetal
Copy link
Author

airmetal commented Nov 22, 2016

Well according to this variables cannot be defined as duplicates anymore.

hashicorp/terraform#8664

@airmetal
Copy link
Author

I never had the dns.tf. At least did not notice it. But then again I have cloned several times so perhaps the last copy did not have it anyway.

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Well according to this variables cannot be defined as duplicates anymore.

Well that sucks. TBH I've been sticking with Terraform v7.1 so I guess I hadn't seen that behaviour.
Fair enough, I'll upgrade to a more recent version and see what breaks.

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

I've fixed the duplicate variable issue @DDDrew - if you fetch the latest code, then you should be good to go.

I'm running a new deploy from scratch here so we'll see how it goes. The SSH key file you're using, is it ~/.ssh/id_rsa.pub? The reason I ask is, otherwise you may have to tell Ansible which key file to use using --key-file=my/key-file.

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

It’s a recent addition (previously it was in a separate branch)…

On 23 Nov 2016, at 9:56 AM, Andrew Das <[email protected]mailto:[email protected]> wrote:

I never had the dns.tf. At least did not notice it. But then again I have cloned several times so perhaps the last copy did not have it anyway.


You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262390958, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkezEb_-KGZHL5sKUEibtqf3W85eX-bks5rA3MmgaJpZM4K4hDS.

itevomcid

@airmetal
Copy link
Author

So on Ansible I removed the known_hosts file and then tried to ssh directly to one of the workers. It then prompted me for the rss key passphrase. After that I ran ansible-playbook and it took off.

What should I have done to prevent this workaround?

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Hmm. I don't know if there's much you can do to prevent it unfortunately (I run into it quite frequently). It's a security feature I think (if you create a host, ssh to it, then destroy and recreate it, then ssh thinks it might be a man-in-the-middle attack since the thumbprint sent by the server that IP no longer matches the one it was expecting).

@airmetal
Copy link
Author

Ok so it’s just a matter of starting with a clean environment. Maybe we can put that in the notes, if that error is encountered?

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Sure sounds good :)

Would you like to open a pull request, or would you rather I did it?

@tintoy
Copy link
Contributor

tintoy commented Nov 22, 2016

Do you still have the exact error text from Ansible? Would be helpful to put it in the docs (we could add a "troubleshooting" section).

@airmetal
Copy link
Author

airmetal commented Nov 22, 2016

Yep here it is. The fix was to remove the known_hosts file and then try to connect again. The first time it prompts for a password and then rest of the hosts are good to go.

Andrews-MacBook-Pro:ansible andrewdas$ ansible-playbook -vvvv playbooks/upgrade-packages.yml 
Using /Users/andrewdas/Documents/Code/Terraform/rancher-ddcloud/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /Users/andrewdas/Documents/Anaconda/anaconda/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: upgrade-packages.yml *************************************************
1 plays in playbooks/upgrade-packages.yml

PLAY [all] *********************************************************************

TASK [clean apt] ***************************************************************
task path: /Users/andrewdas/Documents/Code/Terraform/rancher-ddcloud/ansible/playbooks/upgrade-packages.yml:6
Using module file /Users/andrewdas/Documents/Anaconda/anaconda/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
Using module file /Users/andrewdas/Documents/Anaconda/anaconda/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
Using module file /Users/andrewdas/Documents/Anaconda/anaconda/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<168.128.29.61> ESTABLISH SSH CONNECTION FOR USER: root
Using module file /Users/andrewdas/Documents/Anaconda/anaconda/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<168.128.29.6> ESTABLISH SSH CONNECTION FOR USER: root
<168.128.29.7> ESTABLISH SSH CONNECTION FOR USER: root
<168.128.29.60> ESTABLISH SSH CONNECTION FOR USER: root
<168.128.29.61> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/andrewdas/.ansible/cp/ansible-ssh-%h-%p-%r 168.128.29.61 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-240642740604598 `" && echo ansible-tmp-1479858328.33-240642740604598="` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-240642740604598 `" ) && sleep 0'"'"''
<168.128.29.6> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/andrewdas/.ansible/cp/ansible-ssh-%h-%p-%r 168.128.29.6 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-278029548476700 `" && echo ansible-tmp-1479858328.33-278029548476700="` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-278029548476700 `" ) && sleep 0'"'"''
<168.128.29.7> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/andrewdas/.ansible/cp/ansible-ssh-%h-%p-%r 168.128.29.7 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-50546965555011 `" && echo ansible-tmp-1479858328.33-50546965555011="` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-50546965555011 `" ) && sleep 0'"'"''
<168.128.29.60> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/Users/andrewdas/.ansible/cp/ansible-ssh-%h-%p-%r 168.128.29.60 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-260719869648665 `" && echo ansible-tmp-1479858328.33-260719869648665="` echo $HOME/.ansible/tmp/ansible-tmp-1479858328.33-260719869648665 `" ) && sleep 0'"'"''
fatal: [rancher]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: OpenSSH_6.9p1, LibreSSL 2.1.8\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/Users/andrewdas/.ansible/cp/ansible-ssh-168.128.29.6-22-root\" does not exist\r\ndebug2: ssh_connect: needpriv 0\r\ndebug1: Connecting to 168.128.29.6 [168.128.29.6] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9904 ms remain after connect\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa type 1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519 type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519-cert type -1\r\ndebug1: Enabling compatibility mode for protocol 2.0\r\ndebug1: Local version string SSH-2.0-OpenSSH_6.9\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7\r\ndebug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 168.128.29.6:22 as 'root'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug1: kex: server->client [email protected] <implicit> [email protected]\r\ndebug1: kex: client->server [email protected] <implicit> [email protected]\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug1: Server host key: ecdsa-sha2-nistp256 SHA256:gMQwXUFMlPnnWeDzi2WogThl2Fnlc2ecA+7MGiqO+/w\r\nWarning: Permanently added '168.128.29.6' (ECDSA) to the list of known hosts.\r\ndebug2: set_newkeys: mode 1\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug2: set_newkeys: mode 0\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: SSH2_MSG_SERVICE_REQUEST sent\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug2: key: /Users/andrewdas/.ssh/id_rsa (0x7fb1ab006490),\r\ndebug2: key: /Users/andrewdas/.ssh/id_dsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ecdsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ed25519 (0x0),\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug3: start over, passed a different list publickey,password\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering RSA public key: /Users/andrewdas/.ssh/id_rsa\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_dsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_dsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ecdsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ecdsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ed25519\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ed25519: No such file or directory\r\ndebug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\nPermission denied (publickey,password).\r\n", 
    "unreachable": true
}
fatal: [worker-02]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: OpenSSH_6.9p1, LibreSSL 2.1.8\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/Users/andrewdas/.ansible/cp/ansible-ssh-168.128.29.7-22-root\" does not exist\r\ndebug2: ssh_connect: needpriv 0\r\ndebug1: Connecting to 168.128.29.7 [168.128.29.7] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9890 ms remain after connect\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa type 1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519 type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519-cert type -1\r\ndebug1: Enabling compatibility mode for protocol 2.0\r\ndebug1: Local version string SSH-2.0-OpenSSH_6.9\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7\r\ndebug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 168.128.29.7:22 as 'root'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug1: kex: server->client [email protected] <implicit> [email protected]\r\ndebug1: kex: client->server [email protected] <implicit> [email protected]\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug1: Server host key: ecdsa-sha2-nistp256 SHA256:2Iaql1Hrqs7wrtTu6uwP7iawfXFeI9U0HfLppjoNEyA\r\ndebug3: hostkeys_foreach: reading file \"/Users/andrewdas/.ssh/known_hosts\"\r\nWarning: Permanently added '168.128.29.7' (ECDSA) to the list of known hosts.\r\ndebug2: set_newkeys: mode 1\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug2: set_newkeys: mode 0\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: SSH2_MSG_SERVICE_REQUEST sent\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug2: key: /Users/andrewdas/.ssh/id_rsa (0x7fe18b703450),\r\ndebug2: key: /Users/andrewdas/.ssh/id_dsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ecdsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ed25519 (0x0),\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug3: start over, passed a different list publickey,password\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering RSA public key: /Users/andrewdas/.ssh/id_rsa\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_dsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_dsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ecdsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ecdsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ed25519\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ed25519: No such file or directory\r\ndebug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\nPermission denied (publickey,password).\r\n", 
    "unreachable": true
}
fatal: [worker-01]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: OpenSSH_6.9p1, LibreSSL 2.1.8\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/Users/andrewdas/.ansible/cp/ansible-ssh-168.128.29.61-22-root\" does not exist\r\ndebug2: ssh_connect: needpriv 0\r\ndebug1: Connecting to 168.128.29.61 [168.128.29.61] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9890 ms remain after connect\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa type 1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519 type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519-cert type -1\r\ndebug1: Enabling compatibility mode for protocol 2.0\r\ndebug1: Local version string SSH-2.0-OpenSSH_6.9\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7\r\ndebug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 168.128.29.61:22 as 'root'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug1: kex: server->client [email protected] <implicit> [email protected]\r\ndebug1: kex: client->server [email protected] <implicit> [email protected]\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug1: Server host key: ecdsa-sha2-nistp256 SHA256:Q2peUFNfDkdOIMYMTZDnb/2guA8KwwioJoskw2gyVlo\r\ndebug3: hostkeys_foreach: reading file \"/Users/andrewdas/.ssh/known_hosts\"\r\nWarning: Permanently added '168.128.29.61' (ECDSA) to the list of known hosts.\r\ndebug2: set_newkeys: mode 1\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug2: set_newkeys: mode 0\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: SSH2_MSG_SERVICE_REQUEST sent\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug2: key: /Users/andrewdas/.ssh/id_rsa (0x7fcbb3c10d90),\r\ndebug2: key: /Users/andrewdas/.ssh/id_dsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ecdsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ed25519 (0x0),\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug3: start over, passed a different list publickey,password\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering RSA public key: /Users/andrewdas/.ssh/id_rsa\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_dsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_dsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ecdsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ecdsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ed25519\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ed25519: No such file or directory\r\ndebug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\nPermission denied (publickey,password).\r\n", 
    "unreachable": true
}
fatal: [worker-03]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: OpenSSH_6.9p1, LibreSSL 2.1.8\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/Users/andrewdas/.ansible/cp/ansible-ssh-168.128.29.60-22-root\" does not exist\r\ndebug2: ssh_connect: needpriv 0\r\ndebug1: Connecting to 168.128.29.60 [168.128.29.60] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9890 ms remain after connect\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa type 1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_rsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_dsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ecdsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519 type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /Users/andrewdas/.ssh/id_ed25519-cert type -1\r\ndebug1: Enabling compatibility mode for protocol 2.0\r\ndebug1: Local version string SSH-2.0-OpenSSH_6.9\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7\r\ndebug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 168.128.29.60:22 as 'root'\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: [email protected],zlib,none\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: none,[email protected]\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug1: kex: server->client [email protected] <implicit> [email protected]\r\ndebug1: kex: client->server [email protected] <implicit> [email protected]\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug1: Server host key: ecdsa-sha2-nistp256 SHA256:2tONrsV/nBt2M5NA4z3qLD1MFSoIYZZbNeR0jAP2jwk\r\ndebug3: hostkeys_foreach: reading file \"/Users/andrewdas/.ssh/known_hosts\"\r\nWarning: Permanently added '168.128.29.60' (ECDSA) to the list of known hosts.\r\ndebug2: set_newkeys: mode 1\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug2: set_newkeys: mode 0\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: SSH2_MSG_SERVICE_REQUEST sent\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug2: key: /Users/andrewdas/.ssh/id_rsa (0x7fc129700000),\r\ndebug2: key: /Users/andrewdas/.ssh/id_dsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ecdsa (0x0),\r\ndebug2: key: /Users/andrewdas/.ssh/id_ed25519 (0x0),\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug3: start over, passed a different list publickey,password\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering RSA public key: /Users/andrewdas/.ssh/id_rsa\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_dsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_dsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ecdsa\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ecdsa: No such file or directory\r\ndebug1: Trying private key: /Users/andrewdas/.ssh/id_ed25519\r\ndebug3: no such identity: /Users/andrewdas/.ssh/id_ed25519: No such file or directory\r\ndebug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\nPermission denied (publickey,password).\r\n", 
    "unreachable": true
}
	to retry, use: --limit @/Users/andrewdas/Documents/Code/Terraform/rancher-ddcloud/ansible/playbooks/upgrade-packages.retry

PLAY RECAP *********************************************************************
rancher                    : ok=0    changed=0    unreachable=1    failed=0   
worker-01                  : ok=0    changed=0    unreachable=1    failed=0   
worker-02                  : ok=0    changed=0    unreachable=1    failed=0   
worker-03                  : ok=0    changed=0    unreachable=1    failed=0   

@tintoy
Copy link
Contributor

tintoy commented Nov 23, 2016

Hmm - do you have the non-verbose error?

@airmetal
Copy link
Author

Not anymore.

@tintoy
Copy link
Contributor

tintoy commented Nov 23, 2016

Oh well, no worries. I'm writing something up now.

@tintoy
Copy link
Contributor

tintoy commented Nov 23, 2016

Have a look at the README.

@airmetal
Copy link
Author

May want to indicate that there can be be issue with the known_hosts file as well, especially if executing this after a previous run.

@tintoy
Copy link
Contributor

tintoy commented Nov 23, 2016

Yeah, I was wondering about that :)

The problem is, I don't want to advise people to just delete their known_hosts file if they get an error because that's like telling someone to use HTTP instead of HTTPS because they got a certificate error (i.e. potential security problem). Really I'd rather they actually understood the ramifications of that before proceeding.

On the one hand, I'm trying to avoid turning this into a "how to use SSH" or "how to use Ansible" guide because that sort of thing becomes noise after a while if you already know that and want to know how to troubleshoot more difficult problems. For the most part, I expect anyone who's going to be setting up a Rancher cluster to already know about SSH (if not necessarily Terraform and Ansible).

On the other, I'm not exactly sure of the actual audience for this stuff. Probably not regular developers, but maybe devops people and solution architects?

What do you think - who's the target audience, and depending on that am I perhaps being unreasonable?

@airmetal
Copy link
Author

I think there will be two tracks for this. One developer focused but may still need to be very explicit. We don’t want to limit our audience who is familiar with this stuff. Actually our motive should be reach out to those are not at all but have the technical chops to follow along without wasting time to research. The second track will be more solution architect/manager focused which should really focus on the technologies in use, the architecture and the process flow.

Regards,
Andrew

Andrew Das
Integration Architect (ITAAS) @ Dimension Data
Mobile: +1-919-274-7580
Email: [email protected]

From: Adam Friedman [email protected]
Reply-To: DimensionDataResearch/rancher-ddcloud [email protected]
Date: Tuesday, November 22, 2016 at 7:27 PM
To: DimensionDataResearch/rancher-ddcloud [email protected]
Cc: Andrew Das [email protected], Mention [email protected]
Subject: Re: [DimensionDataResearch/rancher-ddcloud] Issues encountered at various steps (#1)

Yeah, I was wondering about that :)

On the one hand, I'm trying to avoid turning this into a "how to use SSH" or "how to use Ansible" guide because that sort of thing becomes noise after a while if you already know that and want to know how to troubleshoot more difficult problems. For the most part, I expect anyone who's going to be setting up a Rancher cluster to already know about SSH (if not necessarily Terraform and Ansible).

On the other, I'm not exactly sure of the actual audience for this stuff. Probably not regular developers, but maybe devops people and solution architects?

What do you think - who's the target audience, and depending on that am I perhaps being unreasonable?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262406471, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AO81jKIKn8A5q4TdbF7RyK0q6satWeKBks5rA4hUgaJpZM4K4hDS.

itevomcid

@tintoy
Copy link
Contributor

tintoy commented Nov 23, 2016

Ok, it sounds like there's room for a good suite of documentation on this stuff. This repository is not it though; I'd like to see more centralised documentation to improve discoverability :)

Are you interested in helping to write up some tutorials based on the audience you have in mind? I'm happy to review and offer any technical guidance or other assistance where it'd be helpful.

@airmetal
Copy link
Author

Sure, happy to help. Looking for new material for my blog ☺

Regards,
Andrew

Andrew Das
Integration Architect (ITAAS) @ Dimension Data
Mobile: +1-919-274-7580
Email: [email protected]

From: Adam Friedman [email protected]
Reply-To: DimensionDataResearch/rancher-ddcloud [email protected]
Date: Tuesday, November 22, 2016 at 7:36 PM
To: DimensionDataResearch/rancher-ddcloud [email protected]
Cc: Andrew Das [email protected], Mention [email protected]
Subject: Re: [DimensionDataResearch/rancher-ddcloud] Issues encountered at various steps (#1)

Ok, it sounds like theres room for a good suite of documentation on this stuff. This repository is not it though; I'd like to see more centralised documentation to improve discoverability :)

Are you interested in helping to write up some tutorials based on the audience you have in mind? I'm happy to review and offer any technical guidance or other assistance where it'd be helpful.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1#issuecomment-262407885, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AO81jACK8cbSduF3VEHLjN_mIbVkj5RYks5rA4qbgaJpZM4K4hDS.

itevomcid

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants