New datasource: VictoriaLogs

[zekker6]

What would you like to be added?

I would like to have a support of VictoriaLogs (API docs available here) in a way where crowdsec would pull/watch for new log entries similar to an existing integration with Loki.

P.S. I'm one of maintainers of VictoriaLogs and would be happy to create a PR to add the support of this datasource and maintain it later of if needed. Just wanted to check in if you will find this useful before sending a PR.

/kind feature

Why is this needed?

I'm currently pushing all logs coming from different sources to VictoriaLogs and the only way to use crowdsec is to set up a cron task to pull the logs and append these to log lines to a file watched by crowdsec.
This works, but having a datasource support would be make it much cleaner.

[github-actions]

@zekker6: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@zekker6: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[buixor]

Hello,

It sounds like a great idea! Please go ahead, and we'll do our best to review at the appropriate time. Don't hesitate to reach out to us on Discord if needed.

There is no public documentation on how to implement a datasource, however you can look at this interface : https://github.com/crowdsecurity/crowdsec/blob/master/pkg/acquisition/acquisition.go

The Loki datasource should be a good source of inspiration for you too :
https://github.com/crowdsecurity/crowdsec/tree/master/pkg/acquisition/modules/loki

(note we didn't use the tail endpoint of Loki as it was missing logs when there was a burst, but you might not have this issue w/ Victoria).

[tiny-pangolin]

Currently, my solution to this is using telegraf to tail the logs on each sensor, but I would love to see a native API integration, so I don't have to store logs on disk