[Bug] Broker startup could get into a long crash loop when Functions are enabled

[lhotari]

Describe the bug

on pulsar-broker-0, this is the last entry from the logs.
I didn't get a capture of all of the logs, but I could see earlier when it happened, the stacktrace was a Pulsar Admin call, possible related to Pulsar Functions startup.

Caused by: java.net.UnknownHostException: pulsar-broker-2.pulsar-broker.default.svc.cluster.local: Name or service not known
    at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
    at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
    at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
    at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
    at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
    at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
    at io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:169) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:166) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
    at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
    at io.netty.util.internal.SocketUtils.allAddressesByName(SocketUtils.java:166) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.resolver.DefaultNameResolver.doResolveAll(DefaultNameResolver.java:50) ~[io.netty-netty-resolver-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:79) ~[io.netty-netty-resolver-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:71) ~[io.netty-netty-resolver-4.1.100.Final.jar:4.1.100.Final]
    at org.asynchttpclient.resolver.RequestHostnameResolver.resolve(RequestHostnameResolver.java:50) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.request.NettyRequestSender.resolveAddresses(NettyRequestSender.java:357) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:300) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.request.NettyRequestSender.sendNextRequest(NettyRequestSender.java:548) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.handler.intercept.Redirect30xInterceptor.exitAfterHandlingRedirect(Redirect30xInterceptor.java:160) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.handler.intercept.Interceptors.exitAfterIntercept(Interceptors.java:98) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.handler.HttpHandler.handleHttpResponse(HttpHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:140) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-03-21T08:53:41,177+0000 [main] WARN  org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1
java.lang.Exception: Stacktrace for immediate shutdown
    at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:55) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3]
    at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:50) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3]
    at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:82) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3]
    at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:359) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]

To Reproduce

used this command on Docker Desktop integrated k8s:

helm install pulsar apache-pulsar-dist-dev/pulsar --version 3.3.1 --set affinity.anti_affinity=false

It seems that this doesn't happen in every case.

Expected behavior

It shouldn't happen

Desktop (please complete the following information):

• MacOS Apple Silicon M3

Additional Context

• Fixing this requires most likely implementing Create 2 services for brokers: a headless service with publishNotReadyAddresses and an ordinary clusterip service for lookups #437

[lhotari]

I guess the fix would be to have podManagementPolicy: OrderedReady instead of the current podManagementPolicy: Parallel when the Function worker is enabled.