Spike - OpenSearch reporting and notifications assessment

[asteriscos]

Description

We want to identify which of the following functional requirements can be achieved using the OpenSearch Reporting and Notifications plugins. This will allow us to determine the features that will have to be developed in the wazuh-dashboard-plugins repository and how to use the existing features of the mentioned plugins.

• Notifications plugin documentation
• Reporting plugin documentation

Functional Requirements

• Generate PDF reports: The user must be able to generate PDF reports of existing dashboards. This can be achieved by using the OpenSearch reporting plugin.
• Schedule reports: The user must be able to schedule reports within a certain time interval. This can be achieved by using the OpenSearch reporting plugin.
• Send reports by e-mail: The user must be able to receive the scheduled reports by e-mail. Evaluate if we can create a plugin that integrates reports generated in the OpenSearch reporting plugin with the OpenSearch notifications plugin.
• Download reports on demand: The user must be able to download an existing report whenever needed. This can be achieved by using the OpenSearch reporting plugin.
• CRUD reports from one place: Depending on the Wazuh indexer RBAC permissions, the user should be able to manage existing reports and create new ones in one place. This can be achieved by using the OpenSearch reporting plugin.
• Initial threat detection and posture status: Threat detection and posture status will be regularly sent to users via email based on Wazuh dashboard initial startup configuration.

Implementation Restrictions

• Use OpenSearch features: Ensure we use as many native features as possible to achieve the requirements.
• Pre-configured reports: We must be able to pre-configure a set of reports in the initial setup of the application.
• Stateless reports: The generated reports should be stateless to make the docker deployment easier.
• Custom SMTP server: The user must be able to configure its own SMTP server.

Plan

• Analysis
  • Prepare the dev environment to have the reporting and notifications plugins https://quay.io/repository/wazuh/osd-dev?tab=tags 2.14.0.reporting
  • Analyse if the reporting and notifications plugins provides an API to be used from the dashboard
  • Identify RBAC permissions to restrict reporting operations
  • Find a way to generate predefined reports on the initial Wazuh dashboard setup
  • Find a way to generate predefined dashboards on the initial Wazuh dashboard setup
  • Analyze if the OpenSearch notifications plugin provides an interface so we can integrate it with the OpenSearch reporting plugin
  • Analyze if the current Wazuh dashboard's visualizations are compatible with the native features.
  • Analyze if all the desired visualizations are compatible with the reporting system.
• PoC
  • Create a new report alerts plugin that integrates report definitions with the notifications plugin capabilities
  • Create tests dashboards with mocked data as saved objects so they can be referenced in the reporting plugin

Objective

• Reporting revamp #164

Branch

• https://github.com/wazuh/wazuh-dashboard-plugins/tree/enhancement/194-spike-reporting-and-notification-plugins

[yenienserrano]

I was reading the documentation of both plugins and looking for information if they have something to integrate both plugins.
From what I could see, they were going to work on an integration that was going to be in 2.0.0 but apparently it didn't arrive and they don't give much information about when this integration is going to be in 2.0.0.

opensearch-project/reporting#72

On the other hand I saw that you have opensearch-reporting-cli which can be used for sending reports to mail but we would need our dashboards to be added as saved object of dashboards, because you need the id of the visualization.

https://opensearch.org/docs/latest/reporting/rep-cli-create/

And create a development image with the plugins and upload it to quay, to see the capabilities that both plugins provide.

[yenienserrano]

A new branch is created with a new plugin to test and see what can be used of the notification and reporting plugins.

https://github.com/wazuh/wazuh-dashboard-plugins/tree/enhancement/194-spike-reporting-and-notification-plugins

[yenienserrano]

From what I've found out, it seems that they don't have methods to generate reports or send notifications from other plugins.

While testing the applications I saw that the Dashboard application has a button to generate reports using the report plugin.
But when I looked into how they were adding the button to generate the reports I couldn't find it in the visualization plugin.
As far as I can see it is being added by the reporting plugin depending on which application it is in according to the breadcrumb and the buttons added to the top left.

https://github.com/opensearch-project/dashboards-reporting/blob/2.15.0.0/public/components/context_menu/context_menu.js#L264-L316

[jbiset]

Update 2024-06-27

Research about Notifications plugin

To understand the Notifications plugin, it was configured on the platform by configuring a Channel with an SMTP sender through the Resend service, following the OSD guide. Something to keep in mind is that you have to use commands that are not in the UI to load the sender's SMTP service credentials.
Analyzing the functionalities provided by the Notifications plugin, it is likely that the Create Monitor functionality in Alerting (image below) can be used for everything in the schedule; However, it does not solve the problem as to how to generate the report.
It will continue to be analyzed at the code level to evaluate if what was mentioned above can be taken advantage of.

[jbiset]

Update 2024-06-28

• Code from the creation by definition of the Reporting plugin is analyzed
• The possibility of creating reports through Notebooks is analyzed (Documentation)
• Research is being carried out on the Notebooks plugin, currently implicit in the Observability plugin. Related with Issue 195

[yenienserrano]

Update 2024-06-28

• Analysing the plugins in more detail
• Analysing the opensearch-reporting-cli
• Testing opensearch-reporting-cli

[yenienserrano]

Update 2024-07-02

Analysing the creation of a plugin using the stuff from the reporting plugin to create a new plugin with stuff from the reporting plugin and notification plugin

[jbiset]

Update 2024-07-02

• The report generation code is analyzed using Notebooks to analyze the possible development of its own plugin that incorporates the Reporting and Notification functionality.
• Progress is made with reading the documentation in the OSD repository of the internal Notebook plugin within the Observability plugin (plugins/dashboards-observability/public/components/notebooks/docs)
• There is an example of Notebook in plugins/dashboards-observability/public/components/notebooks/docs/example_notebooks/default/Introduction Notebook.json where you can see the structure and specification of the corresponding backend according to the documentation mentioned above.
• We will continue investigating whether this can be useful for the development of our own plugin

[yenienserrano]

Update 2024-07-03

• Adding the reporting plugin to the plugin created to have a base and getting the plugin working.
• Seeing that there are email and notification things in the code that are not being used anywhere.

• Investigating if it was new code from the new integration they want to do and it turns out that it came from the beginning of the plugin that seems to be taken from this repository https://github.com/opendistro-for-elasticsearch/kibana-reports
• Investigating if it is possible to recover some of that code.

[yenienserrano]

Update 2024-07-05

• Configuring the notification plugin to verify that deprecated code uses the notification plugin

• Deprecated code uses the notification plugin to know which channels to send to. 

• Set the test send for the chosen channel.

• Analysing how to add the PDF to the email being sent.

[yenienserrano]

To activate the notification settings I had to add and modify code that was not being used in the reporting plugin, now you can add the channels configured in the notification plugin and do the test sending.

[yenienserrano]

The notification plugin api, can be used in the new reporting plugin

https://opensearch.org/docs/latest/observing-your-data/notifications/api/

[yenienserrano]

Update 2024-07-11

• Investigating how to get the notification channel information
  • Find in the documentation the API that provides the notification plugin.
• Sending email test when running on demand report generation
• Investigating how to run the email test when it is scheduled
• Seeing who is triggering the PDF creation

[yenienserrano]

Apparently the dashboard PDF report generation from what I've seen is generated by the "visualReportId" parameter in the url, which is the ID of the report definition created in the report plugin. So the PDF is only created when the dashboard or notebook is opened in the browser with that parameter.

https://github.com/opensearch-project/dashboards-reporting/blob/main/public/components/context_menu/context_menu.js#L239

[jbiset]

Update 2024-07-11

Although there is no explicit dependency of the Reporting plugin on the Notifications plugin, there is an implicit dependency at the code level, so that if the Notifications plugin is installed, the functionality to configure Notifications for the report is displayed.

Code that checks the Notifications plugin - (plugins/dashboards-reporting/public/components/report_definitions/delivery/delivery.tsx)

Component that is hidden or not according to the previous check

The Delivery component uses the backend services of the reporting plugin server (plugins/dashboards-reporting/server/routes/notifications.ts)

[yenienserrano]

Update 2024-07-12

• Investigating how the PDF is generated by the reporting plugin
  • Apparently it generates it according to what it has in the screen therefore I find complicated to use this to send the report pdf by mail, maybe what is more feasible is to send the URL so that the report is generated but it would be necessary to analyze if it is possible.
    https://github.com/opensearch-project/dashboards-reporting/blob/main/public/components/visual_report/generate_report.ts#L111
• Seeing the settings of the notification plugin in Index Management>Index Management>Notification Settings
  • Investigating how this part is working to better understand the process of sending notifications.

[jbiset]

Update 2024-07-12

Research and testing are being conducted to modify the content of the test email and to attach a file, which in the future will be the report. Although this has not yet been achieved, the specific location where the report content is generated has been identified (plugins/dashboards-reporting/public/components/visual_report/generate_report.ts). This information could be useful for assembling the email attachment. The html2canvas library is used to generate the report.
Regarding the content of the notification email, in the Notifications configuration when creating a report definition, you can configure both the subject and the message to be sent.
The subject is entered via a EuiFieldText input and the message via a markdown editor using the react-mde library.

[jbiset]

Update 2024-07-15

Research continues to be able to edit the content of the Reporting plugin test message through the Notifications plugin. For them, tracking is done from delivery.tsx of the reports plugin to eventRoutes.ts of the Notifications plugin in search of finding the point where the email notification with its testing content is generated.

Notifications plugin path involved

router.post(
    {
      path: `${NODE_API.SEND_TEST_MESSAGE}/{configId}`,
      validate: genericParamsAndDataSourceIdQuery,
    },
    async (context, request, response) => {
      // @ts-ignore
      const client = MDSEnabledClientService.getClient(request, context, dataSourceEnabled);
      try {
        const resp = await client(
          'notifications.sendTestMessage',
          {
            configId: request.params.configId,
          }
        );
        return response.ok({ body: resp });
      } catch (error) {
        return response.custom({
          statusCode: error.statusCode || 500,
          body: error.message,
        });
      }
    }
  );

[yenienserrano]

Update 2024/08/1

I have been getting back on topic and configuring the test environments.
I have been testing the notification plugin in the opensearch management and alerts plugin that is implemented to see if I can find out how they are handling it but still no success.

[yenienserrano]

Update 2024/08/02

Researching and testing the sending of mails. I saw that the configuration of the conditions of the mail sending is saving it in /_plugins/_im/lron as this documentation says, but I still can't find how to send the mail, I left a query in the opensearch slack channel asking if there is any documentation on how to implement the notification plugin.

https://opensearch.org/docs/latest/im-plugin/notifications-settings/

https://opensearch.slack.com/archives/C051D137M7G/p1722613562833819

[yenienserrano]

Update 2024/08/05

It was found that the mail sending is done from the indexer plugin, so it is possible that may need to make changes in the indexer as well.

https://github.com/opensearch-project/notifications/blob/main/notifications/core/src/main/kotlin/org/opensearch/notifications/core/client/EmailMimeProvider.kt

[yenienserrano]

Update 2024/08/13

Investigating a little the scheduled reports that are also made on the side of the indexer, perhaps what can be done at the time that the definition of the report is generated in the indexer since the id is obtained and is part of what is needed to generate the report, it would be necessary to analyze if the other data are also there to be able to build the url

URL:

https://localhost:5601/app/dashboards?security_tenant=global#/view/d2699b30-598a-11ef-a107-db866cece224?_g=(time:(from:'2024-08-13T18:30:13.343Z',to:'2024-08-13T19:00:13.343Z'))&visualReportId=-FEcTZEBgGMgvosQt6uh

Indexer log

https://github.com/opensearch-project/reporting/blob/main/src/main/kotlin/org/opensearch/reportsscheduler/scheduler/ReportDefinitionJobRunner.kt

2024-08-13 21:00:13 [2024-08-13T19:00:13,351][INFO ][o.o.r.s.ReportDefinitionJobRunner] [os1] reports:runJob-created job:-FEcTZEBgGMgvosQt6uh

[yenienserrano]

• To configure a smtp channel in the notification plugin you have to configure the password and username of the sender in the opensearch keystore,
• On the Wazuh Dashboard side we can access the rest of the notification channel configuration through its API.
• Identify in the backend of the reporting plugin indexer where the link to the scheduled PDF report is generated.
• To generate the images and the PDF blob the user has to be connected to the dashboard because that is where the screenshots are taken. We could paste the PDF download link in the body of the email using the indexer plugin (so that the user clicks on the link and can generate the PDF from the UI).
• If we want to use functionalities of the notification plugin it seems impossible to do it without touching things on the indexer side.

[asteriscos]

Example using Reporting CLI

We could also use our own scheduling system that would internally use the reporting CLI.

opensearch-reporting-cli -u https://localhost:5601/app/dashboards#/view/ed7c9360-5bb4-11ef-a9ee-17f85c351195 -a basic -c admin:admin -e smtp -s [email protected] -r [email protected] --smtphost smtp.server.com --smtpport 465 --smtpusername <UserName> --smtppassword <Password> --subject New report available --note New report -f pdf

Docs:

• https://opensearch.org/docs/latest/reporting/rep-cli-install/
• https://github.com/opensearch-project/reporting-cli/blob/main/USER_GUIDE.md

Email HTML template:
https://github.com/opensearch-project/reporting-cli/blob/main/src/views/index.hbs

[yenienserrano]

Environment for modification of the reporting plugin in indexer

1. Clone the reporting repository

https://github.com/opensearch-project/reporting

2. Make the modifications and compile the code with ./gradlew build
   2.1. Have this dockerfile

# Use the OpenSearch base image
FROM opensearchproject/opensearch:2.15.0

# Switch to root user to perform setup
USER root

ENV discovery.type=single-node
ENV OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m"
ENV DISABLE_SECURITY_PLUGIN=true

# Create the plugins directory if it doesn't exist and another directory to place the zip file
RUN mkdir -p /usr/share/opensearch/external-plugins

# Copy the plugin ZIP file into the Docker image with correct ownership
COPY --chown=opensearch:opensearch ./opensearch-reports-scheduler-2.15.0.0-SNAPSHOT.zip /usr/share/opensearch/external-plugins/tasks.zip

# Ensure correct permissions for the plugin ZIP file
RUN chmod 644 /usr/share/opensearch/external-plugins/tasks.zip

# Debug: List the contents of the external-plugins directory to verify the ZIP file is there
RUN ls -la /usr/share/opensearch/external-plugins/

# Debug: Check if the plugin ZIP file exists
RUN if [ -f /usr/share/opensearch/external-plugins/tasks.zip ]; then echo "Plugin ZIP file found"; else echo "Plugin ZIP file not found"; exit 1; fi

# Switch to opensearch user to install the plugin
USER opensearch

# Remove the existing reporting plugin
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-reports-scheduler

RUN ls /usr/share/opensearch/plugins
# Install the plugin from the external-plugins directory
RUN /usr/share/opensearch/bin/opensearch-plugin install file:///usr/share/opensearch/external-plugins/tasks.zip --batch || (echo "Plugin installation failed"; exit 1)

# Expose ports
EXPOSE 9200 9600

# Switch back to root user for any further configuration
USER root

# Debug: Capture the OpenSearch logs
RUN cat /usr/share/opensearch/logs/opensearch.log || echo "Log file not found"

RUN ls /usr/share/opensearch/plugins
# Ensure correct permissions for the plugin directory
RUN chown -R opensearch:opensearch /usr/share/opensearch/plugins/opensearch-reports-scheduler

# Switch back to the opensearch user
USER opensearch

2.2. copy the generated zip (in the build located in the build/distributions folder) at the same level as the dockerfilebuild/distributions

3. Create a docker image: docker build . -t name-image
4. Have an opensearch dashboard with reporting and notifications plugin
5. Create a report and the changes made should be made

[yenienserrano]

Researching and testing the reporting plugin on the indexer side, we were able to modify the plugin to send an email with the notification plugin when the report schedule is generated, testing what could be sent in the email we saw that you can send the email with plain text or HTML.

try sending an email with the report definition link:

try to send an email with the html format

Welcome to Wazuh Dashboard

This is an HTML email!

Patch:

notificationReport.txt

[yenienserrano]

Send report by email

Branch wazuh-dashboard-plugins: enhancement/194-spike-reporting-and-notification-plugins
Patch Reporting plugin: sendEmailHTMLwithKPI.txt

Pre-condition

• Have configured a channel of the notifications plugin

Steps

1. To have a dashboard and create the scheduled report definition.

In the description you can use HTML or plain text.

2. Receive an email when the report schedule interval has passed

You can receive the link to the definition of the created report by adding the text {{urlDefinition}} or the hits with the text {{hits}}.

The hits are taken from a request to the index from the indexer.
The url is built with the data from the report definition and the generated id

Email

Report definition

Report

Dashboard

Discover

Notebooks