Strimzi should allow internal listener to configure brokers[].host

[L1ghtman2k]

Bug Description

I am trying to provision strimzi based operator with istio. I understand that strimzi doesn't support istio, however, in my case, I have to work around this problem because I can only use istio in our environment.

I was able to set-up istio gateway + vs working with strimzi kafka, however, I had to specify listener type: ingress, to allow kafka to configure something on a backend, to make it work with istio. If I try internal listener, get the following error:

Listener configuration is not valid: [listener external cannot configure
        brokers[].host because it is not Route or Ingress based listener, listener
        external cannot configure bootstrap.host because it is not Route or Ingress
        based listener]

Another byproduct of specifying type: ingress, is strimzi creates ingress objects, even though nginx is not on the cluster.

I would like an option that avoids spinning up custom resources, while also allowing external listeners to be exposed by projects other than ingress-nginx.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.40.0

Kubernetes version

1.29

Installation method

Helm Chart

Infrastructure

Bare-metal

Configuration files and logs

No response

Additional context

No response

[scholzj]

The brokers[].host field is used to specify the host in the Kubernetes Ingress or OPenShift Route resources. It does not make any sense to allow it with internal listeners because it would not be used anywhere and would be just ignored anyway.

[L1ghtman2k]

How does bootstrap endpoint know how to advertise broker hostnames to the external clients, if brokers[].host is not specified?

[scholzj]

I have no idea what you think bootstrap endpoint is. Bootstrapping happens by just routing the connection to one of the brokers.

[L1ghtman2k]

I use the following:

import ssl
import logging
from kafka import KafkaAdminClient
from kafka.admin import NewTopic
from kafka.errors import KafkaError, TopicAlreadyExistsError

# Set up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

# Kafka configuration
bootstrap_servers = ['kafka-bootstrap.some.domain:443']
username = 'superuser'
password = '<some-password>'
ca_cert_path = 'ca.pem'  # Path to your existing CA certificate file

# Debug: Print configuration
logger.debug(f"Bootstrap servers: {bootstrap_servers}")
logger.debug(f"Username: {username}")
logger.debug(f"CA Certificate path: {ca_cert_path}")

# Create SSL context with proper security settings
ssl_context = ssl.create_default_context(cafile=ca_cert_path)
ssl_context.check_hostname = True
ssl_context.verify_mode = ssl.CERT_REQUIRED

# Debug: Print SSL context information
logger.debug(f"SSL Context - Check hostname: {ssl_context.check_hostname}")
logger.debug(f"SSL Context - Verify mode: {ssl_context.verify_mode}")


logger.info("Attempting to create Kafka admin client...")
# Create Kafka admin client with SCRAM-SHA-512
admin_client = KafkaAdminClient(
    bootstrap_servers=bootstrap_servers,
    security_protocol='SASL_SSL',
    sasl_mechanism='SCRAM-SHA-512',
    sasl_plain_username=username,
    sasl_plain_password=password,
    ssl_context=ssl_context,
)
logger.info("Kafka admin client created successfully.")

# Debug: Print admin client configuration
logger.debug(f"Admin client config: {admin_client.config}")

# Create a new topic
topic_name = "test_topicdfdd"
logger.info(f"Attempting to create topic: {topic_name}")
new_topic = NewTopic(name=topic_name, num_partitions=1, replication_factor=1)
admin_client.create_topics([new_topic])
logger.info(f"Topic '{topic_name}' created successfully.")

# List all topics
logger.info("Listing all topics...")
topics = admin_client.list_topics()
logger.info(f"Available topics: {topics}")



logger.info("Script execution completed.")

My client figures out that it needs to reach out to one of the brokers by their public hostname, even though, i have not specified any of that in my code. My assumption is that the kafka-bootstrap endpoint returns the public address to the client, so that client can connect to it

[scholzj]

No, it connects to any of the Kafka brokers and it gives it the list of brokers There is no special bootstrap endpoint (or at least I never heard or saw this term being used). But if you want to configure the advertised addresses, you can configure them using the advertised host / port options in the listener configuration. You do not need to change the Ingress or Route host for it.

[L1ghtman2k]

I see, I will try that, thank you!

[L1ghtman2k]

Thanks Jakub, I was able to setup istio with cluster-ip, by configuring advertisedHost & advertisedPort:

      - authentication:
          type: scram-sha-512
        configuration:
          brokerCertChainAndKey:
            certificate: certificate
            key: privateKey
            secretName: infra
          brokers:
          - advertisedHost: kafka-broker0.mydomain.net
            advertisedPort: 443
            broker: 0
        name: external
        port: 9094
        tls: true
        type: cluster-ip     

I then setup the istio gw:

  kind: Gateway
  metadata:
    annotations:
    name: kafka-gateway
    namespace: kafka
  spec:
    selector:
      istio: minio-ingressgateway
    servers:
    - hosts:
      - kafka-broker0.mydomain.net
      - kafka-bootstrap.mydomain.net
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        mode: PASSTHROUGH

and a virtualservice:

  kind: VirtualService
  metadata:
    annotations:
    name: kafka-ingress
    namespace: kafka
  spec:
    gateways:
    - kafka-gateway
    hosts:
    - kafka-bootstrap.mydomain.net
    - kafka-broker0.mydomain.net
    tls:
    - match:
      - port: 443
        sniHosts:
        - kafka-bootstrap.mydomain.net
      route:
      - destination:
          host: kafka-kafka-external-bootstrap
          port:
            number: 9094
    - match:
      - port: 443
        sniHosts:
        - kafka-broker0.mydomain.net
      route:
      - destination:
          host: kafka-kafka-0
          port:
            number: 9094