diff --git a/docs/en/serverless/images/green-dot-icon.png b/docs/en/serverless/images/green-dot-icon.png new file mode 100644 index 0000000000..c886bda449 Binary files /dev/null and b/docs/en/serverless/images/green-dot-icon.png differ diff --git a/docs/en/serverless/images/logs-dataset-overview.png b/docs/en/serverless/images/logs-dataset-overview.png new file mode 100644 index 0000000000..f437ed6d3a Binary files /dev/null and b/docs/en/serverless/images/logs-dataset-overview.png differ diff --git a/docs/en/serverless/images/red-dot-icon.png b/docs/en/serverless/images/red-dot-icon.png new file mode 100644 index 0000000000..95cbb1e77b Binary files /dev/null and b/docs/en/serverless/images/red-dot-icon.png differ diff --git a/docs/en/serverless/images/yellow-dot-icon.png b/docs/en/serverless/images/yellow-dot-icon.png new file mode 100644 index 0000000000..be8e61f66c Binary files /dev/null and b/docs/en/serverless/images/yellow-dot-icon.png differ diff --git a/docs/en/serverless/logging/log-monitoring.mdx b/docs/en/serverless/logging/log-monitoring.mdx index bb9a42d5ba..12969f2b07 100644 --- a/docs/en/serverless/logging/log-monitoring.mdx +++ b/docs/en/serverless/logging/log-monitoring.mdx @@ -79,6 +79,13 @@ The following resources provide information on viewing and monitoring your logs: - Discover and explore: Discover and explore all of the log events flowing in from your servers, virtual machines, and containers in a centralized view. - Detect log anomalies: Use ((ml)) to detect log anomalies automatically. +## Monitor data sets + +The **Data Set Quality** page provides an overview of your data sets and their quality. +Use this information to get an idea of your overall data set quality, and find data sets that contain incorrectly parsed documents. + +Monitor data sets + ## Application logs Application logs provide valuable insight into events that have occurred within your services and applications. diff --git a/docs/en/serverless/logging/monitor-datasets.mdx b/docs/en/serverless/logging/monitor-datasets.mdx new file mode 100644 index 0000000000..87b1ff5d2f --- /dev/null +++ b/docs/en/serverless/logging/monitor-datasets.mdx @@ -0,0 +1,64 @@ +--- +id: serverlessObservabilityMonitorDatasets +slug: /serverless/observability/monitor-datasets +title: Monitor log data set quality +description: Monitor log data sets to find degraded documents. +tags: [ 'serverless', 'observability', 'how-to' ] +--- + +

+

+ +The **Data Set Quality** page provides an overview of your log data sets. +Use this information to get an idea of your overall log data set quality and find data sets that contain incorrectly parsed documents. +Access the Data Set Quality page from the main ((kib)) menu and go to **Stack Management** → **Data Set Quality**. + + + + + Users with the `viewer` role can view the Data Sets Quality summary. To view the Active Data Sets and Estimated Data summaries, users need the `monitor` [index privilege](((ref))/security-privileges.html#privileges-list-indices) for the `logs-*-*` index. + + +The quality of your data sets is based on the percentage of degraded documents in each data set. +A degraded document in a data set contains the [`_ignored`](((ref))/mapping-ignored-field.html) property because one or more of its fields were ignored during indexing. +Fields are ignored for a variety of reasons. +For example, when the [`ignore_malformed`](((ref))/mapping-ignored-field.html.html) parameter is set to true, if a document field contains the wrong data type, the malformed field is ignored and the rest of the document is indexed. + +From the data set table, you'll find information for each data set such as its namespace, when the data set was last active, and the percentage of degraded docs. +The percentage of degraded documents determines the data set's quality according to the following scale: + +* Good (): 0% of the documents in the data set are degraded. +* Degraded (): Greater than 0% and up to 3% of the documents in the data set are degraded. +* Poor (): Greater than 3% of the documents in the data set are degraded. + +Opening the details of a specific data set shows the degraded documents history, a summary for the data set, and other details that can help you determine if you need to investigate any issues. + +## Investigate issues +The Data Set Quality page has a couple of different ways to help you find ignored fields and investigate issues. +From the data set table, you can expand a data set's details, and view commonly ignored fields and information about those fields. +You can also open a data set in Logs Explorer to find ignored fields in individual logs. + +### Find ignored fields in data sets +To expand the details of a dataset with poor or degraded quality and view ignored fields: + +1. From the data set table, click next to a data set with poor or degraded quality. +1. From the details, scroll down to **Degraded fields**. + +The **Degraded fields** section shows fields that have been ignored, the number of documents that contain ignored fields, and the timestamp of last occurrence of the field being ignored. + +### Find ignored fields in individual logs +To use Logs Explorer to find ignored fields in individual logs: + +1. Find data sets with degraded documents using the **Degraded Docs** column of the data sets table. +1. Click the percentage in the **Degraded Docs** column to open the data set in Logs Explorer. + +The **Documents** table in Logs Explorer is automatically filtered to show documents that were not parsed correctly. +Under the *actions* column, you'll find the degraded document icon (). + +Now that you know which documents contain ignored fields, examine them more closely to find the origin of the issue: + +1. Under the **actions** column, click to open the log details. +1. Select the **JSON** tab. +1. Scroll towards the end of the JSON to find the `ignored_field_values`. + +Here, you'll find all of the `_ignored` fields in the document and their values, which should provide some clues as to why the fields were ignored. \ No newline at end of file diff --git a/docs/en/serverless/logging/view-and-monitor-logs.mdx b/docs/en/serverless/logging/view-and-monitor-logs.mdx index 80bcf84caf..95ef68ddba 100644 --- a/docs/en/serverless/logging/view-and-monitor-logs.mdx +++ b/docs/en/serverless/logging/view-and-monitor-logs.mdx @@ -77,4 +77,10 @@ The following actions help you filter and focus on specific fields in the log de * **Filter for value ():** Show logs that contain the specific field value. * **Filter out value ():** Show logs that do _not_ contain the specific field value. * **Filter for field present ():** Show logs that contain the specific field. -* **Toggle column in table ():** Add or remove a column for the field to the main Logs Explorer table. \ No newline at end of file +* **Toggle column in table ():** Add or remove a column for the field to the main Logs Explorer table. + +## View log data set details + +From the main ((kib)) menu, go to **Stack Management** → **Data Set Quality* to view more details about your data sets and monitor their overall quality. + +Refer to Monitor data sets for more information. \ No newline at end of file diff --git a/docs/en/serverless/serverless-observability.docnav.json b/docs/en/serverless/serverless-observability.docnav.json index 98163e6c65..0afbbd8ceb 100644 --- a/docs/en/serverless/serverless-observability.docnav.json +++ b/docs/en/serverless/serverless-observability.docnav.json @@ -65,6 +65,11 @@ "classic-sources": ["enObservabilityMonitorLogs"], "classic-skip": true }, + { + "slug": "/serverless/observability/monitor-datasets", + "classic-sources": ["enObservabilityMonitorDatasets"], + "classic-skip": true + }, { "slug": "/serverless/observability/run-log-pattern-analysis", "classic-sources": ["enKibanaRunPatternAnalysisDiscover"]