Kafka MirrorMaker2 error : Not authorized to access topics: [mm2-offset-syncs.kafka-target.internal]

[fsouza-bux]

Hi guys,

We are testing kafka mirror maker 2 with the following config:

- Source - REMOTE kafka in a different GCP project
- Target - our kafka deployed with strimzi (version 0.21.1)
- Connector / Mirror config : in the target cluster with strimzi.

We bumped into the following error now after configuring:

org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [mm2-offset-syncs.kafka-target.internal]
2021-03-02 13:51:46,851 WARN [Consumer clientId=consumer-null-4, groupId=null] Error while fetching metadata with correlation id 328 : {mm2-offset-syncs.kafka-target.internal=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient) [task-thread-bi-infra-dev-source->kafka-target.MirrorCheckpointConnector-0]
2021-03-02 13:51:46,852 ERROR [Consumer clientId=consumer-null-4, groupId=null] Topic authorization failed for topics [mm2-offset-syncs.kafka-target.internal] (org.apache.kafka.clients.Metadata) [task-thread-bi-infra-dev-source->kafka-target.MirrorCheckpointConnector-0]

1. Does this offset topic needs to created in the source or target kafka? Taking into account that our connector is running in the target.
2. What configuration do we need to apply on creating this topic?
3. What else would need to be pushed to the source kafka?
   4. Is there any way to avoid the creation of this topic on source, but instead on target ?

We don't have direct access to the source kafka, so we would need to ask the maintainers to create anything and I would rather to it at once.

In digging i found some reference saying that the topic should be present in the source kafka it seems (#3375):

This may be due to data in the mm2-offset-syncs.my-target-cluster-1.internal topic on your source cluster that is created and used by the MM2 MirrorSourceConnector. That topic is used to store the offset position that has been reached by the consumer in the MirrorSourceConnector for each mirrored topic-partition. It also stores the mapping of the offset position on the source cluster to the equivalent offset position on the target cluster.

Ps: kafka-target is the alias in the configuration to our kafka.

Thanks

[scholzj]

I do not have an answer to what will be all the ACLs it will need in the source cluster. But I guess you could use some test cluster first as the source to trace what ACLs it will need. Once it has the right ACLs MM2 will create the topics it self.

I know this question came up in the past already, but I do not think anyone shared the list of ACLs needed (maybe people just give it super user instead). So if you figure it out it would be great if you can share it back.

[fsouza-bux]

Hi Jakub,

Thanks for the reply!
We will share here for sure.
One question though, as you mention MM2 will create the topics on the fly, when deployed, i could not see, and not sure how this really works, but it might be not just a case of adding an ACL, but using a superuser, able as well to create this topic on the fly?
I guess we can create the topic ahead and just give permissions to the user used in the mirror to access it as well right?
Thanks,

[scholzj]

One question though, as you mention MM2 will create the topics on the fly, when deployed, i could not see, and not sure how this really works, but it might be not just a case of adding an ACL, but using a superuser, able as well to create this topic on the fly?

In my case it always created them on its own. But as I said, I never looked into the details.

I guess we can create the topic ahead and just give permissions to the user used in the mirror to access it as well right?

In general yes, that should be possible. But I'm not entirely sure:

• What should be the configuration
• Whether it still tries to create it and fails or whether it really doesn't need the ACLs if the topic exists -> you would need to try it I guess

[fsouza-bux]

In my case it always created them on its own. But as I said, I never looked into the details.

Ok, could you tell me in which cases you observed that, and with which user / permissions ? Probably with an admin user or so.

[scholzj]

I do not remember exactly, but probably with super users or without authorization. But super user is just a different form of ACL ... with a test cluster you should be able to figure out what exactly it needs.

From a slightly different perspective ... you should probably also think what do you want. MM2 has a lot of functionality which MM1 dis not have. But you need to think about what are your requirements. If you do not have any access to the source cluster ... do you really need checkpoints? Do you need heartbeats? If not, you can disable them a save some ACLs. You can also consider using MM1 which is really just a consumer and producer and should not need any rights a regular consumer would not need.

[fsouza-bux]

Hi Jakub, thanks again for all the info. I was chasing to use MM2 for being a new version way with improvements as far as i checked, but indeed our use case is to be more of a consumer -> producer in certain cases. In some cases we want to make use of the converters as well, and I'm not sure if this is present in MM1. Yes, regarding checkpoints, it might be a configuration we can dismiss. Thanks again for all the info!