[UI Feature] Support for ELK in Frontend UI Task Log Links and Configuration for Different Log Links Across Multiple Data Plane Clusters

[daadc]

Motivation: Why do you think this is important?

• Centralized Logging: As our system scales, having a centralized logging solution like ELK allows us to aggregate logs across services and infrastructure. This makes it easier to monitor, search, and analyze operational issues or perform advanced troubleshooting.
• Consistency: Users depend on consistent access to logs for debugging workflows and tasks. Having a unified interface that supports ELK will provide a seamless experience to users accustomed to using Kibana for log analysis.
• Multi-Cluster Support: With multiple data plane clusters, each potentially serving different regions or serving as separate environments (development, staging, production), it's important to direct users to the correct logging resources based on where their tasks are executed.

Goal: What should the final outcome look like, ideally?

When a task is submitted to a specific cluster for execution, the corresponding Pod logs will be collected into the ELK logging platform and stored in different indexes based on the data source cluster. In this case, we hope to understand whether Flyte UI can support task log viewing across multiple clusters, which allows users to directly jump to the index corresponding to the task cluster on the ELK platform when viewing task logs.

Reference: https://docs.flyte.org/en/latest/user_guide/productionizing/configuring_logging_links_in_the_ui.html#datadog-integration

Propose: Link/Inline OR Additional context

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[kumare3]

This is no different than stack driver - please recommend a fix and a PR. We will point you tot he code happy to take the contributions