Add SASL authentication for Kafka acquisition

[acm-073]

What would you like to be added?

/kind enhancement
Hello,

crowdsec already supports log acquisition from Kafka streams. I tried to hook it up to Azure Event Hub, which has a Kafka endpoint. There are two things that need to be done to make this work:

• Azure Event Hub requires SASL authentication, which is supported by the underlying kafka-go client but not available in the Kafka acquisition config
• Azure Event Hub requires a TLS connection, but no client certificate. The way the current kafka TLS config works is: either TLS config is present, then a client certificate is expected/required, or TLS config is not present, then a plain socket connection is established

It would be great if crowdsec would support log acquisition from Azure Event Hubs because it is a common scenario in Azure to stream logs through an event hub.

I have created a working prototype and would be happy to create a PR if you're interested.

Regards
Albrecht

Why is this needed?

Streaming logs to Azure Event Hub is a common scenario in Azure cloud architectures. Being able to acquire logs directly from Event Hubs would greatly simplify the integration of crowdsec with Azure cloud applications.

[github-actions]

@acm-073: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@acm-073: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[acm-073]

/kind enhancement

[LaurenceJJones]

With this

Azure Event Hub requires a TLS connection, but no client certificate. The way the current kafka TLS config works is: either TLS config is present, then a client certificate is expected/required, or TLS config is not present, then a plain socket connection is established

Do you want to just just load the CA certificate then just so I can understand? as if the ca cert is within the system, then you shouldnt need to define it here unless you can tell us if it self signed as we dont know how azure does its TLS.

Edit: ahh I think I see the issue.

[acm-073]

With this

Azure Event Hub requires a TLS connection, but no client certificate. The way the current kafka TLS config works is: either TLS config is present, then a client certificate is expected/required, or TLS config is not present, then a plain socket connection is established

Do you want to just just load the CA certificate then just so I can understand? as if the ca cert is within the system, then you shouldnt need to define it here unless you can tell us if it self signed as we dont know how azure does its TLS.

Edit: ahh I think I see the issue.

Let me detail some more:

• if the TLS config in kafka acquisition is present, then the acquisition tries to load a client cert which it would use for client authentication. If none is configured, then the client configuration fails. A client cert is NOT required for Azure Event Hub - only server authentication (where the server presents a SSL certificate) is used. For this, of course the client needs to have the "public" CA certificates, but they are usually available in the environment
• if the TLS config is not present, the acquisition tries to establish a plain tcp connection without using TLS, which fails with Azure Event Hubs.

My approach was to add a SASL config like this:

sasl:
  mechanism: PLAIN
  username: $ConnectionString
  password: <azure-eventhub-connection-string>
  use_ssl: true

and then in Kafka Dialer, set the TLS config either if kafka.TLS is set (with the client cert stuff) or if kafka.sasl.use_ssl is set, without client cert.

You can have a look at acm-073@cb500a1#diff-69964165ec35fb39e2763740aa78ff24fdd4f24055edadd9fb61d31d7ea29e7cR282 to see how I solved the issue.

Regards
Albrecht