Spike - OpenSearch Observability plugin and Wazuh metrics assessment

[asteriscos]

Description

For the next major release of Wazuh, we want to incorporate metrics and traces about the different components of Wazuh in the dashboard.

To achieve this, we want to leverage the OpenSearch Observability plugin, as it provides a framework to work with metrics and traces. This framework is works with OpenTelemetry, which will be used in other Wazuh components.

The goal of this issue is to identify the capabilities and restrictions of the OpenSearch Observability plugin to generate Wazuh metrics and traceability reports. Within the observability plugin lies the Notebooks application which allows the enhancement of standard dashboards with code snippets, live visualizations, and narrative text.
These Notebooks can be used to generate complex reports.

We need to:

• determine if we can use the observability plugin it to manage the metrics and traces of Wazuh.
• evaluate and design the events format, as well as dashboards and visualizations covered by the plugin.
• evaluate the capabilities of the notebooks plugin to explore and report Wazuh observability metrics

References:

• Observability plugin documentation
• Notebooks documentation
• OpenSearch and OpenTelemetry
• OpenTelemetry documentation
• https://medium.com/@erikstrm_11611/demystify-opentelemetry-with-opensearch-as-a-single-plane-of-glass-919f884eb568
• Observability plugin documentation

Functional Requirements

• Generate PDF reports: The user must be able to generate PDF reports of engine and agent comms metrics.
• Initial threat detection and posture status: Threat detection and posture status will be regularly sent to users via email based on Wazuh dashboard initial startup configuration.

Implementation Restrictions

• Use OpenSearch features: Ensure we use as many native features as possible to achieve the requirements.
• Pre-configured reports: We must be able to pre-configure a set of reports in the initial setup of the application.
• Stateless reports: The generated reports should be stateless to make the docker deployment easier.

Plan

• Analysis
  • Prepare the dev environment to have the Observability plugin
  • Detect restrictions and features to be developed in wazuh-dashboard-plugins
  • Identify RBAC permissions to restrict operations in the Observability plugin
  • Identify events format requirements and limitations
  • Identify a way to have initial dashboards in Wazuh dashboards initial setup
• PoC
  • Generate mock metrics using the open telemetry protocol
  • Create a dashboard in the Notebooks app using the mocked metrics
  • Generate PDF reports

Objective

• Reporting revamp #164

[jbiset]

Update 2024-07-03

The code of the Observability plugin is investigated and analyzed focusing on the idea of using the Notebook
Below is a rendering flowchart to get to the Notebooks CRUD page.

Something to note that may be interesting is that they use DashboardContainerByValueRenderer, similar to the current rendering of Dashboards

[jbiset]

Update 2024-07-04

Notebook rendering tracking is deepened.
Below is the flow and format of the Input of the DashboardContainerByValueRenderer that ends up being used in a Notebook when it is a visualization.

Flowchart from the Notebook main

OutputBody VISUALIZATION code case

case 'VISUALIZATION':
        let from = moment(visInput?.timeRange?.from).format(dateFormat);
        let to = moment(visInput?.timeRange?.to).format(dateFormat);
        from = from === 'Invalid date' ? visInput.timeRange.from : from;
        to = to === 'Invalid date' ? visInput.timeRange.to : to;
        return (
          <>
            <EuiText size="s" style={{ marginLeft: 9 }}>
              {`${from} - ${to}`}
            </EuiText>
            <DashboardContainerByValueRenderer
              key={key}
              input={visInput}
              onInputUpdated={setVisInput}
            />
          </>
        );

visInput

{
    "viewMode": "view",
    "panels": {
        "1": {
            "gridData": {
                "x": 0,
                "y": 0,
                "w": 48,
                "h": 20,
                "i": "1"
            },
            "type": "visualization",
            "explicitInput": {
                "id": "1",
                "savedObjectId": "c6182e90-3a46-11ef-9824-2bce77daa33d"
            }
        }
    },
    "isFullScreenMode": false,
    "filters": [],
    "useMargins": false,
    "id": "ie5b34eb1-3a46-11ef-b2f0-61dbd60e329b",
    "visSavedObjId": "c6182e90-3a46-11ef-9824-2bce77daa33d",
    "timeRange": {
        "to": "2024-07-04T20:49:10.429Z",
        "from": "2024-06-04T20:49:10.429Z"
    },
    "title": "embed_viz_ie5b34eb1-3a46-11ef-b2f0-61dbd60e329b",
    "query": {
        "query": "",
        "language": "lucene"
    },
    "refreshConfig": {
        "pause": true,
        "value": 15
    }
}

Note

Although in this case inputVis uses a savedObjectId, in theory it should also support the definition of a visualization as currently used in the different dashboards.

[jbiset]

Update 2024/07/05

Based on what was previously investigated, progress is made in testing whether a Notebook can render the visualizations of the panels of one of the current dashboards, in this case, Threat Hunting, using the definition of the visualizations through savedVis instead of using a savedObjectId. Based on this, it is confirmed that whether it is savedVis or savedObjectId, a Notebook will be able to render both depending on the visInput that is passed to it.

Original Threat Hunting screen

Panels with visualizations of the Threat Hunting center in a Notebook

[jbiset]

Update 2024-07-08

Based on the Notebook CRUD components, it is observed that they use an API to create and store data. What's more, when loading notebook examples, the API is used to generate the Notebooks, as well as the corresponding visualizations are generated at the moment to assign them to the Notebooks' paragraphs.

Taking into account the following files we can see how the Notebooks are managed:

• plugins/dashboards-observability/public/components/notebooks/components/main.tsx
• plugins/dashboards-observability/public/components/notebooks/components/notebook.tsx
• plugins/dashboards-observability/public/components/notebooks/components/paragraph_components/para_output.tsx

Below is the method OSD uses to add the Notebooks example

addSampleNotebooks = async () => {
    try {
      this.setState({ loading: true });
      const flights = await this.props.http
        .get('../api/saved_objects/_find', {
          query: {
            type: 'index-pattern',
            search_fields: 'title',
            search: 'opensearch_dashboards_sample_data_flights',
          },
        })
        .then((resp) => resp.total === 0);
      const logs = await this.props.http
        .get('../api/saved_objects/_find', {
          query: {
            type: 'index-pattern',
            search_fields: 'title',
            search: 'opensearch_dashboards_sample_data_logs',
          },
        })
        .then((resp) => resp.total === 0);
      if (flights || logs) this.setToast('Adding sample data. This can take some time.');
      await Promise.all([
        flights ? this.props.http.post('../api/sample_data/flights') : Promise.resolve(),
        logs ? this.props.http.post('../api/sample_data/logs') : Promise.resolve(),
      ]);
      const visIds: string[] = [];
      await this.props.http
        .get('../api/saved_objects/_find', {
          query: {
            type: 'visualization',
            search_fields: 'title',
            search: '[Logs] Response Codes Over Time + Annotations',
          },
        })
        .then((resp) => visIds.push(resp.saved_objects[0].id));
      await this.props.http
        .get('../api/saved_objects/_find', {
          query: {
            type: 'visualization',
            search_fields: 'title',
            search: '[Logs] Unique Visitors vs. Average Bytes',
          },
        })
        .then((resp) => visIds.push(resp.saved_objects[0].id));
      await this.props.http
        .get('../api/saved_objects/_find', {
          query: {
            type: 'visualization',
            search_fields: 'title',
            search: '[Flights] Flight Count and Average Ticket Price',
          },
        })
        .then((resp) => visIds.push(resp.saved_objects[0].id));
      await this.props.http
        .post(`${NOTEBOOKS_API_PREFIX}/note/addSampleNotebooks`, {
          body: JSON.stringify({ visIds }),
        })
        .then((res) => {
          const newData = res.body.map((notebook: any) => ({
            path: notebook.name,
            id: notebook.id,
            dateCreated: notebook.dateCreated,
            dateModified: notebook.dateModified,
          }));
          this.setState((prevState) => ({
            data: [...prevState.data, ...newData],
          }));
        });
      this.setToast(`Sample notebooks successfully added.`);
    } catch (err: any) {
      this.setToast('Error adding sample notebooks.', 'danger');
      console.error(err.body.message);
    } finally {
      this.setState({ loading: false });
    }
  };

How to create a Notebook

// Creates a new notebook
  createNotebook = (newNoteName: string) => {
    if (newNoteName.length >= 50 || newNoteName.length === 0) {
      this.setToast('Invalid notebook name', 'danger');
      window.location.assign('#/');
      return;
    }
    const newNoteObject = {
      name: newNoteName,
    };

    return this.props.http
      .post(`${NOTEBOOKS_API_PREFIX}/note`, {
        body: JSON.stringify(newNoteObject),
      })
      .then(async (res) => {
        this.setToast(`Notebook "${newNoteName}" successfully created!`);
        window.location.assign(`#/${res}`);
      })
      .catch((err) => {
        this.setToast(
          'Please ask your administrator to enable Notebooks for you.',
          'danger',
          <EuiLink href={NOTEBOOKS_DOCUMENTATION_URL} target="_blank">
            Documentation
          </EuiLink>
        );
        console.error(err);
      });
  };

How to load a Notebook

loadNotebook = () => {
   this.showParagraphRunning('queue');
   this.props.http
     .get(`${NOTEBOOKS_API_PREFIX}/note/` + this.props.openedNoteId)
     .then(async (res) => {
       this.setBreadcrumbs(res.path);
       let index = 0;
       for (index = 0; index < res.paragraphs.length; ++index) {
         // if the paragraph is a query, load the query output
         if (res.paragraphs[index].output[0]?.outputType === 'QUERY') {
           await this.loadQueryResultsFromInput(res.paragraphs[index]);
         }
       }
       this.setState(res, this.parseAllParagraphs);
     })
     .catch((err) => {
       this.props.setToast(
         'Error fetching notebooks, please make sure you have the correct permission.',
         'danger'
       );
       console.error(err);
     });
 };

How to add a paragraph in a Notebook

  addPara = (index: number, newParaContent: string, inpType: string) => {
    const addParaObj = {
      noteId: this.props.openedNoteId,
      paragraphIndex: index,
      paragraphInput: newParaContent,
      inputType: inpType,
    };

    return this.props.http
      .post(`${NOTEBOOKS_API_PREFIX}/paragraph/`, {
        body: JSON.stringify(addParaObj),
      })
      .then((res) => {
        const paragraphs = [...this.state.paragraphs];
        paragraphs.splice(index, 0, res);
        const newPara = this.parseParagraphs([res])[0];
        newPara.isInputExpanded = true;
        const parsedPara = [...this.state.parsedPara];
        parsedPara.splice(index, 0, newPara);

        this.setState({ paragraphs, parsedPara });
        this.paragraphSelector(index);
        if (this.state.selectedViewId === 'output_only')
          this.setState({ selectedViewId: 'view_both' });
      })
      .catch((err) => {
        this.props.setToast(
          'Error adding paragraph, please make sure you have the correct permission.',
          'danger'
        );
        console.error(err);
      });
  };

Paragraph rendering based on type

if (typeOut !== undefined) {
    switch (typeOut) {
      case 'QUERY':
        const inputQuery = para.inp.substring(4, para.inp.length);
        const queryObject = JSON.parse(val);
        if (queryObject.hasOwnProperty('error')) {
          return <EuiCodeBlock key={key}>{val}</EuiCodeBlock>;
        } else {
          const columns = createQueryColumns(queryObject.schema);
          const data = getQueryOutputData(queryObject);
          return (
            <div>
              <EuiText key={'query-input-key'} className="wrapAll" data-test-subj="queryOutputText">
                <b>{inputQuery}</b>
              </EuiText>
              <EuiSpacer />
              <QueryDataGridMemo
                key={key}
                rowCount={queryObject.datarows.length}
                queryColumns={columns}
                dataValues={data}
              />
            </div>
          );
        }
      case 'MARKDOWN':
        return (
          <EuiText
            key={key}
            className="wrapAll markdown-output-text"
            data-test-subj="markdownOutputText"
          >
            <MarkdownRender source={val} />
          </EuiText>
        );
      case 'VISUALIZATION':
        let from = moment(visInput?.timeRange?.from).format(dateFormat);
        let to = moment(visInput?.timeRange?.to).format(dateFormat);
        from = from === 'Invalid date' ? visInput.timeRange.from : from;
        to = to === 'Invalid date' ? visInput.timeRange.to : to;
        return (
          <>
            <EuiText size="s" style={{ marginLeft: 9 }}>
              {`${from} - ${to}`}
            </EuiText>
            <DashboardContainerByValueRenderer
              key={key}
              input={visInput}
              onInputUpdated={setVisInput}
            />
          </>
        );
      case 'OBSERVABILITY_VISUALIZATION':
        let fromObs = moment(visInput?.timeRange?.from).format(dateFormat);
        let toObs = moment(visInput?.timeRange?.to).format(dateFormat);
        fromObs = fromObs === 'Invalid date' ? visInput.timeRange.from : fromObs;
        toObs = toObs === 'Invalid date' ? visInput.timeRange.to : toObs;
        const onEditClick = (savedVisualizationId: string) => {
          window.location.assign(`observability-logs#/explorer/${savedVisualizationId}`);
        };
        return (
          <>
            <EuiText size="s" style={{ marginLeft: 9 }}>
              {`${fromObs} - ${toObs}`}
            </EuiText>
            <div style={{ height: '300px', width: '100%' }}>
              <VisualizationContainer
                http={getOSDHttp()}
                editMode={false}
                visualizationId={''}
                onEditClick={onEditClick}
                savedVisualizationId={para.visSavedObjId}
                pplService={getPPLService()}
                fromTime={para.visStartTime}
                toTime={para.visEndTime}
                onRefresh={false}
                pplFilterValue={''}
                usedInNotebooks={true}
              />
            </div>
          </>
        );
      case 'HTML':
        return (
          <EuiText key={key}>
            {/* eslint-disable-next-line react/jsx-pascal-case */}
            <Media.HTML data={val} />
          </EuiText>
        );
      case 'TABLE':
        return <pre key={key}>{val}</pre>;
      case 'IMG':
        return <img alt="" src={'data:image/gif;base64,' + val} key={key} />;
      default:
        return <pre key={key}>{val}</pre>;
    }
  } else {
    console.log('output not supported', typeOut);
    return <pre />;
  }

TO DO: Analyze how to combine the use of Notifications Channels to send a report based on a notebook

[yenienserrano]

Update 2024-08-08

Researching Traces indexes for view operation and added scripts to generate sample data in that view.

To the branch enhancement/194-spike-reporting-and-notification-plugins

[yenienserrano]

Update 2024-08-09

investigating what indexes were allowed for the Observability > Metrics view, I could find in the code that it is filtering for indexes with the name ss4o_metrics-*-*.

https://github.com/opensearch-project/dashboards-observability/blob/main/common/constants/metrics.ts#L48

Then I tried to add documents and I had problems with the template so I investigated the fields and the format of each field and I found this file where it said so.

https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/metrics/metrics.md

so I started building the template and testing the examples that are in the same folder. (Then I found that the templates are assembled here)

[yenienserrano]

• 2 index pattern are required for the trace view to work: otel-v1-apm-span-* and v1-apm-service-map* for data prepper or *jaeger-span-* and *jaeger-service* for jaeger https://opensearch.org/docs/latest/observing-your-data/trace/trace-analytics-jaeger/#about-data-ingestion-with-jaeger-indexes
• Metrics with data prepper you need the index pattern to be ss4o_metrics-*-* because it is hardcoded in a constant.
• The trace visualizations are already defined and we cannot edit or use those visualizations in other notebooks or dashboards (so you could not generate PDF reports).
• The metrics visualizations and the logs>visualizations can be used in the notebooks.
• The notebooks report uses the reporting plugin, if it is not installed it does not show the button to generate reports https://github.com/opensearch-project/dashboards-observability/blob/main/public/components/notebooks/components/notebook.tsx#L989C49-L989C75

[yenienserrano]

Branch wazuh-dashboard-plugin: enhancement/194-spike-reporting-and-notification-plugins
Repository Data prepper: https://github.com/opensearch-project/data-prepper.git

1. Clone Data prepper repository
2. in the Wazuh-dashboard-plugins repository go to the branch: enhancement/194-spike-reporting-and-notification-plugins
3. Modify the path of the data prepper volumes
4. Running the development environment
5. On port 8089 is the application to execute the sample data.
6. Initialize dashboard yarn start --no-base-path
7. In traces you should have information
8. Add indece in dashboards managements to view it in Discover

Opensearch repository of a demo of opentelemetry: https://github.com/opensearch-project/opentelemetry-demo/tree/main

Metrics fields and samples: https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/metrics/metrics.md

Metrics configuration blog: https://opensearch.org/blog/opentelemetry-metrics-visualization/#visualizations-with-tsvb