Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Stream naming is far too restrictive #744

Open
morrone opened this issue Jan 19, 2024 · 1 comment
Open

Data Stream naming is far too restrictive #744

morrone opened this issue Jan 19, 2024 · 1 comment

Comments

@morrone
Copy link

morrone commented Jan 19, 2024

The approach to Data Stream naming in the connector is too restrictive. Right now the only option we have is to pattern the data stream name like so:

<data.stream.type>-<data.stream.dataset>-<topic>

This pattern doesn't even really allow us to match the common data steam naming scheme which, in kafka terms, would look a bit more like:

<type>-<topic>-<namespace>

And further, the data.stream.type can only be "metrics" or "logs".

None of these restrictions exist in elasticsearch itself; they are only imposed by the connector.

We should really be able to specify the entire data stream name, using some basic substitution variables (for topic name, etc.) in the string.

@rz1027
Copy link

rz1027 commented Apr 22, 2024

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants