--domain flag didn't work

[RichardBradley]

I am trying to deploy Concourse with a domain name

I ran:

control-tower deploy --iaas AWS myci.example.com --domain myci.example.com

The deploy succeeded, but it has set "concourse-url" to an ip address.
Concourse is reachable at that address.

I would like it to use myci.example.com instead.

There are no obvious errors in the output.

How can I debug?

Thanks,

Rich

[irbekrm]

Hey Rich,

I think the issue might be that all the flags need to be before the actual deployment name, i.e control-tower deploy --iaas AWS --domain myci.example.com myci.example.com
If it had picked up the domain flag, you should be able to see an A record for myci.example.com in example.com zone.

Hopefully this will work!

[RichardBradley]

Thanks for your help, that fixed it.

This surprises me; I would not expect unixy tools to insist on flag v.s. command order like that (c.f. grep [1]), or at least to complain if I get it wrong (c.f. git grep [2])

[1]:

$ echo foo | grep bar --invert-match
foo
$ echo foo | grep --invert-match bar
foo

[2]:

$ git grep foo --invert-match
fatal: option '--invert-match' must come before non-option arguments

Is this behaviour something from go's command line parser?

[irbekrm]

Glad it worked!
We use urfave/cli go package to define commands, flags, etc, so I think this behaviour comes from how this package works.

[RichardBradley]

After fixing that, my credhub is broken:

$ ./credhub find
Get https://ci.example.com:8844/info: x509: certificate is not valid for any names, but wanted to match ci.example.com

Do I have to destroy and re-create the Concourse, or is there a way to get control-tower to fix just the credhub?

I think the problem is that the credhub was deployed when I didn't pass an --domain arg, and hasn't been updated since I re-ran with a --domain arg.

I tried control-tower maintain --renew-nats-cert, but that didn't fix it.

[irbekrm]

I'm not sure whether it's a Credhub or cert issue.
What shell are you using? We had some issues before where in zsh shell the exported CREDHUB_CA_CERT was in wrong format, but worked fine in bash.

[RichardBradley]

What shell are you using? We had some issues before where in zsh shell the exported CREDHUB_CA_CERT was in wrong format, but worked fine in bash.

Bash. I have been able to connect to credhub fine before for instances created by control-tower.

The problem here, I think, is that I created this credhub instance using control-tower without a --domain (which causes credhub to use a self-signed cert on an IP address, I think), then updated it to use a domain with control-tower, and it's now in an intermediate broken state.

[irbekrm]

Destroying and deploying again seems like a good idea, let us know if it works.

control-tower maintain --renew-nats-cert is for rotating NAT certificates for BOSH (for BOSH's internal use), it wouldn't fix an issue with credhub certs.

[RichardBradley]

I deleted and re-created the server back in Nov and it worked.

Now, I have changed the domain name for the instance by re-running the "control-tower deploy" command and it is stuck in the same intermediate broken state.

The instance is up at the new hostname, but none of the jobs work.

If I try to connect to the credhub, I get:

$ credhub export -f 2020-05-15-ci-secrets.yaml
Get https://new-name:8844/info: x509: certificate is valid for old-name, not new-name

Do I have to delete + re-create again? I would really like to find some way to fix this without doing that, as I now have a lot of history.

[RichardBradley]

I tried the "renew-https-cert" job but it did nothing as the main cert has many days left.

I tried bosh recreate and it did a load of stuff but I still have the same problem.

The cert on the credhub is signed by "/C=USA/O=Cloud Foundry/CN=ConcourseCA". How can I rebuild it?

I tried the "self-update" job, but it is stuck, likely due to the credhub issue

[RichardBradley]

I tried deleting the "web" machine, which holds the bad cert. BOSH re-created it after a short pause, but with the wrong cert again.

I am going to delete and re-create the cluster :-(

[RichardBradley]

I did this again :-(

I was deleting and recreating my cluster due to an irrecoverable cert issue (see #334), then I created my cluster without specifying "domain" on first run and I needed to delete + recreate it again, as credhub becomes stuck in this "certificate is not valid for any names" state