Issue accessing rabbitmq amqps tls port outside of AKS cluster

[sajithvasu]

We are facing issues with rabbitmq running 3 replica set deployed on AKS platform using this chart https://charts.bitnami.com/bitnami where amqps port access outside of the AKS cluster fails with "connection reset by peer" error.

Below are the details:

• Deploy RabbitMQ on AKS from chart https://charts.bitnami.com/bitnami
• Services are exposed using kubernetes svc type as "LoadBalancer" and external dns record has been associated with the public IP.
• Non-TLS port 5672 works fine through Public IP when exposed using Loadbalancer.
• management console access is done using nginx ingress resource with letsencrypt so we don't had to worry about tls for management port.
• For amqps TLS port 5671, self signed certificate created with multiple SAN ( following * *.-nodes..svc. * ..svc.) for each rabbit mq node on server side as per the document - https://www.rabbitmq.com/kubernetes/operator/using-operator.html#one-way-tls
• Created client certificate using the same CA and ensured common name matches with the user on rabbitmq.
• Chart configuration for tls setup:

tls:
enabled: true
autoGenerated: false
failIfNoPeerCert: true
sslOptionsVerify: verify_peer
caCertificate: |-
serverCertificate: |-
serverKey: |-
existingSecret: "rabbitmq-certs"
existingSecretFullChain: false

extraConfiguration: |-
ssl_cert_login_from = common_name
auth_backends.1 = rabbit_auth_backend_internal
auth_backends.2 = rabbit_auth_backend_amq
auth_mechanisms.1 = PLAIN
auth_mechanisms.2 = EXTERNAL

extraPlugins: "rabbitmq_auth_backend_ldap rabbitmq_auth_mechanism_ssl"

• RabbitMQ instances are running fine on AKS with no error but when we try to test using sample python pika program running outside of cluster, then it fails with error "connection reset by peer"
• I don't see any error in the rabbitmq logs to troubleshoot further and it seems like traffic is not routed to the service pod. But Telnet works fine for port 5671.
• Also, Tried tls-gen as well to create certificate but no luck.

Error on the client side:

~/cloud/sample-program$ /home/user/.pyenv/versions/3.9.7/bin/python /home/user/cloud/sample-program/send-ssl.py
INFO:pika.adapters.utils.connection_workflow:Pika version 1.0.0 connecting to ('x.x.x.x', 5671)
INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('x.x.x.x', 48930), raddr=('x.x.x.x', 5671)>
ERROR:pika.adapters.utils.io_services_utils:SSL do_handshake failed: error=ConnectionResetError(104, 'Connection reset by peer'); <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('x.x.x.x', 48930)>
Traceback (most recent call last):
File "/home/user/.pyenv/versions/3.9.7/lib/python3.9/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/home/user/.pyenv/versions/3.9.7/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
ERROR:pika.adapters.utils.connection_workflow:Attempt to create the streaming transport failed: ConnectionResetError(104, 'Connection reset by peer'); 'x.x.x.x'/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('x.x.x.x', 5671)); ssl=True
ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorTransportSetupError: ConnectionResetError(104, 'Connection reset by peer')
ERROR:pika.adapters.utils.connection_workflow:AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: ConnectionResetError(104, 'Connection reset by peer'); first exception - None.
ERROR:pika.adapters.utils.connection_workflow:AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: ConnectionResetError(104, 'Connection reset by peer'); first exception - None
ERROR:pika.adapters.base_connection:Full-stack connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: ConnectionResetError(104, 'Connection reset by peer'); first exception - None
ERROR:pika.adapters.base_connection:Self-initiated stack bring-up failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: ConnectionResetError(104, 'Connection reset by peer'); first exception - None
INFO:pika.connection:AMQP stack terminated, failed to connect, or aborted: error-arg=AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: ConnectionResetError(104, 'Connection reset by peer'); first exception - None; pending-error=None
INFO:pika.connection:Connection setup terminated due to AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: ConnectionResetError(104, 'Connection reset by peer'); first exception - None
ERROR:pika.callback:Calling <bound method Connection._default_on_connection_error of <SelectConnection CLOSED transport=None params=>> for "0:_on_connection_error" failed

No error seen on rabbitmq logs.

Here we are not using rabbitmq K8s cluster operator. Any help here will be appreciated.