Access Talos API through an external NGINX load balancer

[flombois]

Hi all,

I am trying out Talos Linux (v1.4.3) using Hyper-V on a Windows server.

I am currently attempting to setup a single-node cluster (for now) following this documentation https://www.talos.dev/v1.4/talos-guides/install/virtualized-platforms/hyper-v but I have an issue to push the machine configuration.

The control plane node is running behind a NGINX reverse proxy that we have set as a load balancer

upstream control-planes-talos {
        ip_hash;
        server cp0-srv0-dc0.local.example.com:50000;
}


server {
        listen  80;
        server_name cluster0.example.com;
        return 301 https://$host$request_uri;
}

server {
        listen  443 ssl http2;
        server_name cluster0.example.com;

        ssl_certificate /etc/nginx/certificates/example.com/fullchain.pem;
        ssl_certificate_key /etc/nginx/certificates/example.com/privkey.pem;

        ssl_protocols TLSv1.3 TLSv1.2;
        ssl_ecdh_curve secp384r1;
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-G$
        ssl_prefer_server_ciphers on;

        ssl_trusted_certificate /etc/nginx/certificates/example.com/fullchain.pem;
        ssl_stapling on;
        ssl_stapling_verify on;

        add_header Strict-Transport-Security "max-age=15552000; includeSubdomains; preload";

        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 5m;
        ssl_session_tickets off;

        location / {
                allow 192.168.0.0/26;
                allow 172.20.1.6;
                deny all;
                proxy_read_timeout 1800;
                proxy_connect_timeout 1800;
                proxy_send_timeout 1800;
                send_timeout 1800;
                proxy_pass http://control-planes-talos;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header X-Nginx-Proxy true;
                proxy_redirect off;
        }
}

The load balancer dns is cluster0.example.com and the control plane node assigned address is 172.20.1.6 so I run:

user@localhost:$ talosctl apply-config --insecure  --talosconfig=./talosconfig --file=./controlplane.yaml --nodes 172.20.1.6
error applying new configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 172.20.1.6:50000: i/o timeout"

I am using this talosconfig file:

context: cluster0
contexts:
    cluster0:
        endpoints:
            - https://cluster0.example.com:443
        ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJQakNCOGFBREFnRUNBaEJvV2F6WFEvanUycDN2Mk1FWmV4TzRNQVVHQXl0bGNEQVFNUTR3REFZRFZRUUsKRXdWMFlXeHZjekFlRncweU16QTNNVEF4TnpReU1qQmFGdzB6TXpBM01EY3hOelF5TWpCYU1CQXhEakFNQmdOVgpCQW9UQlhSaGJHOXpNQ293QlFZREsyVndBeUVBWFh6K0tEU3ZqTm5LcmdiSXBpYmpEQ3hubThUVTREakpZeG9ZCjgzd3lONUtqWVRCZk1BNEdBMVVkRHdFQi93UUVBd0lDaERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUkKS3dZQkJRVUhBd0l3RHdZRFZSMFRBUUgvQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVL2dlQ1UxbmgvTk4xVU9hawo0eWhjeWJqbStFRXdCUVlESzJWd0EwRUFPVzhjTUp1cGtVTkNXN2ZzZjE0NnJIUytLbGY4bEN3cDQ1YVI2Skl0Cmc5Y2JaMzh5dDI3RkxkUUFBQ1FISzZTdHhxNUxuNkNIN2QwUlV3a2h4dWdkQUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
        crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJLRENCMjZBREFnRUNBaEFWQ0xPT1k2b0t2b2lNQmNpeHVZTG5NQVVHQXl0bGNEQVFNUTR3REFZRFZRUUsKRXdWMFlXeHZjekFlRncweU16QTNNVEF4TnpReU1qQmFGdzB6TXpBM01EY3hOelF5TWpCYU1CTXhFVEFQQmdOVgpCQW9UQ0c5ek9tRmtiV2x1TUNvd0JRWURLMlZ3QXlFQWVpaXh3MVVrMEVqdUp2TmtIQ2EvQk95M1ZPR08zSWZLCnNHSGhiODEvMEdTalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJSGdEQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0QKQWpBZkJnTlZIU01FR0RBV2dCVCtCNEpUV2VIODAzVlE1cVRqS0Z6SnVPYjRRVEFGQmdNclpYQURRUUR3SlBIQgpqenFCWGQxb1BxdmxIZnZ4ZDUyRWQxQ3pPMGlYZTN1K3pyWXFsWGhCOXFabUlteEZzWUxxQk13WlRQU3k3LzR0CnFhcWhRR1VkS3pRVG1MUUsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
        key: LS0tLS1CRUdJTiBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0KTUM0Q0FRQXdCUVlESzJWd0JDSUVJTDBUNXdvdzdRb2E2MGFiMVRqNXJlTS9lZzNmbG9XVVUyYkQ1c25jbWNLbwotLS0tLUVORCBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0K

The DNS server is configured to resolve cp0-srv0-dc0.local.example.com to 172.20.1.6, the control plane node.

I expect to have the talosctl API calls sent to the NGINX load balancer on port 443 as set as the endpoint in talosconfig and then the request being forwarded to one of the control plane node (currently only one: 172.20.1.6) but it seems to timeout.

We configured the node network using the UI in the virtual machine interface:

I have tried a few other commands but they all timed out, when omitting -i flag returns a 502 bad gateway but I think this is expected.
I haven't yet configured the load balancer for the kubernetes api server (port 6443) to focus on troubleshooting this first error, does they need both (talosctl API server and kubernetes API server) to be configured to be able to bootstrap.

May you also confirm that the talosctl API server is reachable on port 50000 as mentioned here https://www.talos.dev/v1.4/introduction/getting-started/#decide-how-to-access-the-talos-api

Maybe, if you have some examples, can you share an external NGINX load balancer configuration for Talos please ?

Let me know what I may have missed or misunderstood,

Thank you very much.

[smira]

Talos API uses gRPC and mutual TLS, it can't be proxied via NGINX. You can set up a pure TCP load-balancer if you can't access your nodes directly.

Kubernetes API needs TCP load-balancer as well.

[flombois]

Hi @smira

Thanks a lot for your quick reply,

May you explain why it can't be proxied through NGINX please ? Because of your reply, we have tried to set up NGINX as a TCP load balancer as well: https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer with this configuration on port 50000:

stream {
        upstream control-planes-talos {
                server cp0-srv0-dc0.local.example.com:50000;
        #       server cp1-srv0-dc0.local.example.com:50000 down;
        }

        server {
                listen  50000 ssl;

                ssl_certificate /etc/nginx/certificates/example.com/fullchain.pem;
                ssl_certificate_key /etc/nginx/certificates/example.com/privkey.pem;

                ssl_protocols TLSv1.3 TLSv1.2;
                ssl_ecdh_curve secp384r1;
                ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA->
                ssl_prefer_server_ciphers on;

                ssl_trusted_certificate /etc/nginx/certificates/example.com/fullchain.pem;

                allow 192.168.0.0/26;
                allow 172.20.1.6;
                deny all;
                proxy_pass control-planes-talos;
        }
}

It seems that NGINX may also support gRPC: https://nginx.org/en/docs/http/ngx_http_grpc_module.html we currently use the latest stable version of NGINX (v1.24.0)

What is a "pure" TCP load balancer ? Is it possible that SSL/TLS configuration is wrong and if so how can I quickly verify it ? The request to 172.20.1.6 (the control plane) keep timing out when using the --insecure flag. Without the flag the command fails:

error getting container list: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

So I think this is maybe because the SSL/TLS config is wrong but I am not sure how to verify it because I don't see any message being log in the talos VM.

If its definitely not possible to use NGNIX as a load balancer (not a reverse proxy) which load balancer would you recommend that works best with talos/k8s ?

Thanks a lot for your help,

Regards

[smira]

There are ways to build more complicated solutions with impersonation, etc., but that is not easy and requires full understanding of the authentication and authorization. I would say it's very advanced case.

Talos API is using mutual TLS (both client and server certificates are verified), so TLS termination should be done in the Talos node, which implies that load balancer should simply pass through TCP connections without doing any TLS termination/HTTP/gRPC handling.

I'm not an expert in the ways to configure NGINX for that purpose, but all you need is simple TCP load balancing without any TLS termination.E.g. something like haproxy should be configurable for that purpose.

[legege]

Thanks to the grpc-go library, talosctl does take into account the HTTPS_PROXY environment variables. Thus, another approach that works really well is to deploy a forward proxy (e.g. Squid, tinyproxy, etc.) that does have access to your Talos nodes. It works transparently.