TLS doesn't work: no viable challenge type found

[a3kov]

I'm hosting an app on GCP using a free domain from DuckDNS. Ports 80 and 443 are open.
After successful deploy accessing the server on https:// doesn't work. The logs contain this:

2024-09-27T03:20:06.170255841Z {"time":"2024-09-27T03:20:06.169664821Z","level":"INFO","msg":"http: TLS handshake error from 1.2.3.4:47452: acme/autocert: unable to satisfy \"https://acme-v02.api.letsencrypt.org/acme/authz-v3/1111111111\" for domain \"mydomain.duckdns.org\": no viable challenge type found"}

Proxy config in deploy.yml

proxy:
  ssl: true
  app_port: 3000
  host: mydomain.duckdns.org,api.mydomain.duckdns.org
  healthcheck:
    path: /health

[eavsievich]

Having the same issue with a Cloudflare DNS. It was working previously with Traefik though.

Edited: the solution was to temporarily disable Proxy mode for the domain in Cloudflare.

[kevinmcconnell]

The TLS-ALPN-01 challenge we're using requires a direct connection on port 443. Cloudflare proxying will prevent that from working. I'm not familiar with DuckDNS but if there is any intermediate step happening there, it could be a similar problem.

I think we can work around this, though, just by allowing the HTTP-01 challenge type, and ensuring that Cloudflare (or whomever) is forwarding HTTP traffic. I'd like to look more closely at the tradeoffs before adding this, but I'll test it out and see where we land.

[elct9620]

I think making Kamal Proxy set a default CA for the Cloudflare case may resolve the issue. We can use Origin CA generated by Cloudflare so that the proxy can connect to Kamal Proxy correctly.

I disabled the proxy to make the ACME challenge pass and enabled it again for now. I find the flexible mode does not work because Kamal Proxy didn't set any default CA that HTTPS is always returns an error until the ACME challenge is passed.

[kevinmcconnell]

@elct9620 when #17 lands you'll have the ability to use the Cloudflare origin certs by explicitly adding them to the proxy. That is one way to get Cloudflare proxying to work in "Full (strict)" mode, although it will require adding those certs yourself. I don't want to add any Cloudflare-specific defaults, but you'll be able to set that up yourself if you want.

But with http-01 I think "Full (strict)" mode should work without any manual setup, as we can provision the certs on the connection from Cloudflare to origin.

[kevinmcconnell]

Got this working in a branch now and it seems like the right solution. Should have something merged for this soon.

[a3kov]

The TLS-ALPN-01 challenge we're using requires a direct connection on port 443. Cloudflare proxying will prevent that from working. I'm not familiar with DuckDNS but if there is any intermediate step happening there, it could be a similar problem.

DuckDNS only provides DNS records, it's not involved in routing requests here. And port 443 is open. I'm not using any other software - it's plain Ubuntu 24.04 server with Kamal. Not using Cloudflare or any other proxy in front of the server. I'm also not using GCP load balancer - it's a single compute instance

[clayton]

I ran into this issue upgrading a Rails 7.2 app to Kamal 2.0.

I typically run cloudflare security on "Full." But only after getting the initial Kamal deploy running (and seeing the Traefik self-signed cert warning) while it's running as "off" in the cloudflare dashboard.

Edit: The temporary fix would be to not proxy the domain in cloudflare DNS and point directly to your host IP

[a3kov]

The Cloudflare issue is not related: I'm not using Cloudflare. Can somebody confirm TLS-ALPN-01 challenge works at all with Kamal-proxy ?

[elct9620]

@a3kov

According to the autocert document, I think it should be supported.
https://pkg.go.dev/golang.org/x/crypto/acme/autocert#Manager.GetCertificate

If GetCertificate is used directly, instead of via Manager.TLSConfig, package users will also have to add acme.ALPNProto to NextProtos for tls-alpn-01, or use HTTPHandler for http-01.

Kamal Proxy also enables the ALPN protocol

kamal-proxy/internal/server/server.go

Lines 97 to 104 in 059bead

	s.httpsServer = &http.Server{
	Addr: httpsAddr,
	Handler: handler,
	TLSConfig: &tls.Config{
	NextProtos: []string{"h2", "http/1.1", acme.ALPNProto},
	GetCertificate: s.router.GetCertificate,
	},
	}

[a3kov]

@elct9620
I mean yes, it obviously should be supported since the docs mention you only need 443 port open to set up SSL. I would like some anecdotal evidence though, because it could be a bug in Kamal-proxy and not an issue in my setup

[elct9620]

I don't know if this discussion on Caddy is the same issue.
https://caddy.community/t/caddy-not-able-to-obtain-tls-certificate-for-duckdns-domain/23438/7

The Let's Encrypt website says Caddy supports TLS-ALPN-01 but it seems somebody has some config issue on it.

---

For Cloudflare Proxy, the "Let's Debug" tool is mentioned in the discussion. It explain the reason Kamal Proxy cannot run with it.

Fatal
The domain my-domain.example.com is being served through Cloudflare CDN, which supports the HTTP & HTTPS protocols only. It is impossible to obtain a certificate using the TLS-ALPN-01 challenge with the Cloudflare CDN proxy enabled.

I think the autocert will fallback to HTTP-01 then Cloudflare Proxy will be unable to reach Kamal Proxy under Full mode.

[a3kov]

Yeah the "Let's Debug" tool gives me failures also.. Could be too picky Let's Encrypt behavior or misbehaving DNS of DuckDNS. Anyway I'll also wait for the HTTP-01 challenge implementation - I don't care which one to use as long as it works in the end

[elct9620]

Anyway, I make a PR #29 to let HTTP-01 challenge available.

[kevinmcconnell]

I've added support for HTTP-01 challenges in v0.7.0. If anyone wants to try it on that version directly, please do. I will get an updated release of Kamal out shortly that uses this version too.

Cloudflare's proxied connections now work out-of-the-box. Set your SSL mode to something like "Full (strict)", and it should set itself up automatically.

Also, just circling back to this part of the issue:

Can somebody confirm TLS-ALPN-01 challenge works at all with Kamal-proxy?

@a3kov It does work, yes. Prior to that last commit, this is the only challenge type we were using. Could you check that the IP DuckDNS is returning for your instance is definitely the public IP of your server, and that you can connect to it from outside your network?

[a3kov]

@kevinmcconnell Thanks for the confirmation. The A record of the domain works - that's how I trigger the TLS challenge verification. Judging from the "Let's Debug" tool behavior TLS-ALPN-01 needs more than A/AAAA records - it complains something about SOA for the zone. Maybe it needs complete ownership of the zone ? No idea..

[a3kov]

"Let's Debug" gave me DNS failures even with http-01 challenge, so I gave up on DuckDNS. Probably some years ago it worked with LetsEncrypt - I can only find old reports. But now it looks like they've tightened the requirements w.r.t. DNS. I've switched to a proper domain (if anybody is interested, nic.us.kg gives you a free 3rd level domain you actually own)

[kevinmcconnell]

@a3kov thanks, it does seem like that's a separate issue with DuckDNS then. I'll look into that a bit later (I'm not sure it's really a Kamal Proxy issue after all, but I'm curious to see where it fails). Thanks for reporting it.

In the meantime, I believe the http-01 support is what was required for the other cases (e.g. Cloudflare), so I'm closing this one. Feel free to open a new issue if you hit anything else!

[emmceemoore]

I was also getting the no viable challenge type found error. I'm using Cloudflare as my proxy and discovered that the challenge wasn't completing successfully because I had a WAF rule in place that blocked non-US traffic. I updated my rules to allow any requests with a "URI Path" that starts with /.well-known/acme-challenge/ through and things started working for me (I noticed that Let's Encrypt was sending requests from Finland and Singapore in addition to ones it was sending from the US).