-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
616b582
commit 59c1649
Showing
4 changed files
with
17 additions
and
13 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,12 +10,15 @@ excerpt: Changes to expect in our logging system in December 2024 | |
|
||
As we [announced on November 21, 2024]({{site.base_url}}{% link _posts/2024-11-21-new-logging-system.md %}), | ||
we are upgrading the Cloud.gov customer application logging system and the user interface | ||
at https://logs.fr.cloud.gov. The application logs interface prior to December 2024 | ||
at <https://logs.fr.cloud.gov>. The application logs interface prior to December 2024 | ||
was based on [_Kibana_](https://www.elastic.co/kibana), and we are migrating to one | ||
based on [_OpenSearch Dashboards_](https://www.opensearch.org/docs/latest/dashboards/). | ||
|
||
While all the underlying functionality is unchanged, or improved, there are some | ||
differences between Kibana (old) and OpenSearch (new), which we'll outline here: | ||
differences between Kibana (old) and OpenSearch (new), which we'll outline here. | ||
|
||
**Note: You will need to [migrate your saved objects](#migrating-kibana-customizations-to-opensearch)** (searches, visualizations) from Kibana | ||
to OpenSearch before January 7, 2025. After that date, importing objects will require a support request. | ||
|
||
## Logging in to the new system | ||
|
||
|
@@ -66,10 +69,13 @@ or as individual `.ndjson` files: | |
|
||
### Importing saved objects into OpenSearch | ||
|
||
**If you used the same saved object in Kibana across multiple Cloud.gov orgs**, | ||
you will need to import it into each OpenSearch tenant (each tenant corresponds to a Cloud.gov Cloud Foundry "org"). | ||
|
||
Once you've exported the objects as `.ndjson` files, switch to OpenSearch, then: | ||
|
||
* From the LeftNav menu, select, "Management -> Dashboards Management" | ||
* On the"Dashboards Management" window, select "Saved Objects" on the left menu | ||
* From the LeftNav menu, select, "Management -> Dashboards Management". | ||
* On the "Dashboards Management" window, select "Saved Objects" on the left menu. | ||
* Then select "Import" on the upper right corner. | ||
* On the "Import saved objects" window: | ||
* Select the file to upload | ||
|
@@ -80,8 +86,6 @@ Once you've exported the objects as `.ndjson` files, switch to OpenSearch, then: | |
* If the import results in an "Overwrite index-pattern" dialog, you will likely want to "Skip" the overwrite: ![Screenshot of Overwrite index-pattern with "Skip" selected]({{site.baseurl}}/assets/images/content/opensearch-import-overwrite-dialog.png) | ||
* When the import is complete, click "Done" | ||
|
||
**If you used the same saved object in Kibana across multiple Cloud.gov orgs**, | ||
you will need to import it into each OpenSearch tenant (org). | ||
|
||
|
||
### Recovering Saved Searches and Visualizations after Kibana decomissioning | ||
|
@@ -98,28 +102,28 @@ The screenshot below show some of the major changes to the user interfaces, such | |
2. The "Top 5 values" for a field view is now an option to the right of the field, instead of a double-click | ||
3. There are a lot more values gathered for container metrics | ||
|
||
![Screenshot comparing Kibana to OpenSearch]({{site.baseurl}}/assets/images/content/opensearch-ui-diffences.png) | ||
![Screenshot comparing Kibana to OpenSearch]({{site.baseurl}}/assets/images/content/opensearch-ui-differences.png) | ||
|
||
## Key system differences | ||
|
||
The Cloud.gov team has implemented OpenSearch to deliver a number of benefits to our customers. Among these are: | ||
|
||
* Twelve months of live access to system logs, in alignment with M-21-31. | ||
* Definitions of saved searches and visualizations are now isolated by Cloud.gov organization. | ||
* Twelve months of live access to system logs, in alignment with [M-21-31](https://www.whitehouse.gov/wp-content/uploads/2021/08/M-21-31-Improving-the-Federal-Governments-Investigative-and-Remediation-Capabilities-Related-to-Cybersecurity-Incidents.pdf). | ||
* Definitions of saved searches and visualizations are now isolated by OpenSearch tenants that correspond to Cloud.gov organizations. | ||
* You no longer need to worry about choosing a globally unique name. | ||
* If you share the same saved object across multiple orgs, you will need to import it into each of your orgs. | ||
* Better handling of large log messages. Both Kibana/ELK and OpenSearch have a 32kb limit on message size. The older system dropped such messages from Kibana (although they were still retained in cold storage), the newer system keeps the first 32kb and discards the rest | ||
* Truncated messages are tagged with `_messagetrimmed`. | ||
* Extremely large log messages (over 1Gb) are trimmed and tagged `_logtrimmed` -- such message are probably indicative of a coding error in your application. | ||
* You can search for such messages with a filter of `@logs is one of "_messagetrimmed", "_logtrimmed"`, as shown here | ||
![Screenshot from OpenSearch edit filter with settings as described above]({{site.baseurl}}/assets/images/content/opensearch-logtrimmed.png) | ||
* AWS Brokered Service Logs (Beta): If your Cloud.gov organization had already arranged Cloudwatch ingest of RDS database logs, then you can filter for those logs with the filter `@version: 1`. Most databases, as of December 2024, are not shipped to Cloudwatch and OpenSearch. | ||
* AWS Brokered Service Logs (Beta): If your Cloud.gov organization had already arranged Cloudwatch ingest of RDS database logs, then you can filter for those logs with the filter `@version: 1`. Most databases, as of December 2024, are not yet shipped to Cloudwatch and OpenSearch. | ||
* Cloud.gov will be expanding the availability and features of RDS logs, and other brokered services, in 2025. This is a [beta feature]({{site.baseurl}}/docs/services/intro/#support-status) and subject to change. | ||
* JSON log parsing: Custom logs are not at risk of being dropped because of index field limits. JSON logs are now ingested using the [flat_object field type](https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/) in OpenSearch. The flat_object field type allows for [searching nested fields of a JSON object](https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/#using-flat-object) using dot notation. | ||
* JSON log parsing: Custom logs are not at risk of being dropped because of index field limits. JSON logs are now ingested using the [flat_object field type](https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/) in OpenSearch. The `flat_object` field type allows for [searching nested fields of a JSON object](https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/#using-flat-object) using dot notation. | ||
* Additional container metrics: We now log additional container metrics, available under the `containermetric.name` field. Particularly useful is the `containermetric.name: cpu_entitlement`, which is a way to track whether you're [exceeding the allowed CPU for your apps](https://www.cloudfoundry.org/blog/better-way-split-cake-cpu-entitlements/). | ||
|
||
## Reporting Issues and Getting Help | ||
|
||
Report any undocumented issues you encounter, or questions you may have, to [email protected]. | ||
Report any undocumented issues you encounter, or questions you may have, to <[email protected]>. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters