Fixing fluentd config for metrics keys to match log output

.github/workflows/slack-notify-issues.yml

@@ -15,6 +15,6 @@ jobs:
           SLACK_COLOR: '#00A86B'
           SLACK_ICON: https://pbs.twimg.com/profile_images/978188446178082817/86ulJdF0.jpg
           SLACK_TITLE: "[${{ github.event.issue.state}}] ${{ github.event.issue.title }} on ${{ github.repository }} :rocket:"
-          SLACK_MESSAGE: 'Link: ${{ github.event.issue.url }}'
+          SLACK_MESSAGE: 'Link: ${{ github.event.issue.html_url }}'
           SLACK_USERNAME: PartnerEngineers
           SLACK_WEBHOOK: ${{ secrets.SLACK_ISSUE_WEBHOOK }}

CHANGELOG.md

@@ -1,68 +1,92 @@
 # JFrog Log Analytics Changelog
+
 All changes to the log analytics integration will be documented in this file.
 
+## [1.0.3] - April 05, 2024
+
+* Adding multi-arch support (amd64, arm64) for the sidecar docker image to allow also deployment in ARM environments
+* Using loki charts instead of the deprecated loki-stack charts
+* Updating Readme installation instructions
+
 ## [1.0.2] - Jan 10, 2024
+
 * Upgrading fluentd image to the latest image from releases.jfrog.io (Fluentd version 1.16.3)
 
 ## [1.0.1] - Nov 14, 2023
+
 * Upgrading fluentd image to the latest image from releases.jfrog.io (Fluentd version 1.15.2)
 * Removing sed substitutions in helm charts to eliminate human errors in loki URL
 
 ## [1.0.0] - Dec 12, 2022
+
 * Prometheus Solution Redesigned
 * Addition of Loki Component for Log Data Ingestion
 * Enhanced Prometheus for Metrics Data Ingestion
 
 ## [0.15.0] - Oct 14, 2022
-* Deprecation notice
 
+* Deprecation notice
 
 ## [0.10.0] - Feb 14, 2022
+
 * Added call home configuration for the artifactory fluent configuration.
 
 ## [0.9.0] - Dec 31, 2020
+
 * Repo layout fixes
 * Support for Helm
 * Readme updates
 
 ## [0.8.0] - Nov 30, 2020
+
 * Bug fixes
 
 ## [0.7.2] - Oct 29, 2020
+
 * Updated docker chart queries to monitor [dockerhub rate limiting](https://jfrog.com/blog/get-around-docker-download-limits-jfrog-artifactory/).
 
 ## [0.7.1] - Oct 28, 2020
+
 * Updated fluentd config, added new chart widgets and an alert to monitor [dockerhub rate limiting](https://jfrog.com/blog/get-around-docker-download-limits-jfrog-artifactory/).
 
 ## [0.7.0] - Oct 20, 2020
+
 * Fixing issue with ip_address in access logs having space and . at the end
 
 ## [0.6.1] - Sept 25, 2020
+
 * Updated Prometheus and Grafana installation to use the Prometheus community kube-prometheus-stack. Added drill down options to charts.
 
 ## [0.6.0] - Sept 25, 2020
+
 * [BREAKING] Prometheus fluentd configs updated to use JF_PRODUCT_DATA_INTERNAL env.
 
 ## [0.5.1] - Sept 9, 2020
+
 * Prometheus repo now submodule of parent log-analytics
 
 ## [0.5.0] - Sept 8, 2020
+
 * [No Prometheus] Adding JFrog Pipelines fluent configuration files to capture logs
 
 ## [0.4.0] - Sept 4, 2020
+
 * [No Prometheus] Adding JFrog Mission Control fluent configuration files to capture logs
 
 ## [0.3.0] - Aug 26, 2020
+
 * [No Prometheus] Adding JFrog Distribution fluent configuration files to capture logs
 
 ## [0.2.0] - Aug 24, 2020
+
 * Splunk updates to launch new version of Splunkbase app v1.1.0
 
 ## [0.1.1] - June 1, 2020
+
 * Removing the need for user to specify splunk host , user, and token twice
 * Fixing issue with regex on the audit security log
 * Fixed issue with the repo and image when not docker api url
 
 ## [0.1.0] - May 12, 2020
-* Initial release of Jfrog Logs Analytic integration
 
+* Initial release of Jfrog Logs Analytic integration

README.md

@@ -23,17 +23,17 @@ The JFrog Log Analytics and Metrics solution using Prometheus consists of three
    1. For Installation and usage refer [HELM Guide](https://helm.sh/docs/intro/install/)
 4. Versions supported and Tested:
 
-   Jfrog Platform: 10.9.2
+   Jfrog Platform: 10.17.3
 
-   Artifactory : 7.46.10
+   Artifactory : 7.77.8
 
-   Xray : 3.59.4
+   Xray : 3.92.7
 
-   Prometheus:2.39.1
+   Prometheus 2.51.0
 
-   Grafana:9.2.3
+   Grafana:10.4.0
 
-   Loki: 2.6.1
+   Loki: 2.9.6
 
 ## Read me before installing
 
@@ -159,7 +159,9 @@ Then helm install the Artifactory or Artifactory HA charts:
 
 Artifactory ⎈:
 
-helm install `artifactory` chart (using the above generated join and master keys). Note that you need to be at the root of this repository folder to have the helm values yaml available
+helm install `artifactory` chart (using the above generated join and master keys).
+
+**💡Note: You need to be at the root of this repository folder to have `helm/artifactory-values.yaml` file available for the following command**
 
 ```bash
 helm upgrade --install artifactory jfrog/artifactory \
@@ -173,7 +175,9 @@ helm upgrade --install artifactory jfrog/artifactory \
 
 Artifactory-HA ⎈:
 
-helm install `artifactory-ha` chart (using the above generated join and master keys). Note that you need to be at the root of this repository folder to have the helm values yaml available
+helm install `artifactory-ha` chart (using the above generated join and master keys).
+
+**💡Note: You need to be at the root of this repository folder to have `helm/artifactory-ha-values.yaml` file available for the following command**
 
 ```bash
 helm upgrade --install artifactory-ha jfrog/artifactory-ha \
@@ -183,7 +187,7 @@ helm upgrade --install artifactory-ha jfrog/artifactory-ha \
        -n $INST_NAMESPACE
 ```
 
-**If you are installing in the same cluster with the deprecated solution, Use the same namespace as the previous one instead of jfrog-plg above.**
+**💡Note: If you are installing in the same cluster with the deprecated solution, Use the same namespace as the previous one instead of jfrog-plg above.**
 Note the above examples are only references you will need additional parameters to configure TLS, binary blob storage, or other common Artifactory features.
 
 This will complete the necessary configuration for Artifactory and expose a new service monitor `servicemonitor-artifactory` to expose metrics to Prometheus.
@@ -200,7 +204,9 @@ Generate master keys for the Xray installation:
 export XRAY_MASTER_KEY=$(openssl rand -hex 32)
 ````
 
-Use the same `JOIN_KEY` from the Artifactory installation, in order to connect Xray to Artifactory. Note, that you need to be at the root of this repository folder to have the `helm/xray-values.yaml` available
+Use the same `JOIN_KEY` from the Artifactory installation, in order to connect Xray to Artifactory.
+
+**💡Note: You need to be at the root of this repository folder to have `helm/xray-values.yaml` file available for the following command**
 
 ```bash
 # getting Artifactory URL
@@ -218,6 +224,8 @@ helm upgrade --install xray jfrog/xray --set xray.jfrogUrl=http://$RT_SERVICE_IP
 
 **If you are installing in the same cluster with the deprecated solution, Use the same namespace as the previous one instead of jfrog-plg above.**
 
+# Configuration
+
 ## To Assess the setup for Prometheus
 
 Use 'kubectl port forward' as mentioned below in a separate terminal window
@@ -270,7 +278,7 @@ After downlowding the dashboards go to "Dashboards" -> "Import":
 
 Pick `Uplaod JSON file` and upload Artifactory and Xray dashboards files that you downloaded in the previous step.
 
-Import the Dashboards and select the appropriate sources
+Import the Dashboards and select the appropriate sources (Loki and Prometheus)
 
 ![dashboards](images/DashboardAddition-02.jpeg)
 

fluent_metrics.conf.xray

@@ -87,7 +87,7 @@
     name sys_memory_free_bytes
     desc System Memory Free in Bytes
     type gauge
-    key sys_memory_free_bytes
+    key _sys_memory_free_bytes
     <labels>
       agent fluentd
       host ${hostname}
@@ -99,7 +99,7 @@
     name app_disk_free_bytes
     desc Application Disk Free in Bytes
     type gauge
-    key app_disk_free_bytes
+    key _app_disk_free_bytes
     <labels>
       agent fluentd
       host ${hostname}
@@ -111,7 +111,7 @@
     name app_disk_used_bytes
     desc Application Disk Used in Bytes
     type gauge
-    key app_disk_used_bytes
+    key _app_disk_used_bytes
     <labels>
       agent fluentd
       host ${hostname}
@@ -123,7 +123,7 @@
     name go_memstats_heap_objects_total
     desc GO Memory Statistics Total Heap Objects
     type gauge
-    key go_memstats_heap_objects_total
+    key _go_memstats_heap_objects_total
     <labels>
       agent fluentd
       host ${hostname}
@@ -135,7 +135,7 @@
     name go_memstats_heap_allocated_bytes
     desc GO Memory Statistics Heap Allocated in Bytes
     type gauge
-    key go_memstats_heap_allocated_bytes
+    key _go_memstats_heap_allocated_bytes
     <labels>
       agent fluentd
       host ${hostname}
@@ -147,7 +147,7 @@
     name go_memstats_heap_in_use_bytes
     desc GO Memory Statistics Heap Used in Bytes
     type gauge
-    key go_memstats_heap_in_use_bytes
+    key _go_memstats_heap_in_use_bytes
     <labels>
       agent fluentd
       host ${hostname}
@@ -159,7 +159,7 @@
     name go_memstats_heap_reserved_bytes
     desc GO Memory Statistics Heap Reserved in Bytes
     type gauge
-    key go_memstats_heap_reserved_bytes
+    key _go_memstats_heap_reserved_bytes
     <labels>
       agent fluentd
       host ${hostname}

helm/artifactory-ha-values.yaml

@@ -23,7 +23,7 @@ artifactory:
           name: volume
   customSidecarContainers: |
     - name: "artifactory-fluentd-sidecar"
-      image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.1"
+      image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.2"
       imagePullPolicy: "IfNotPresent"
       volumeMounts:
         - mountPath: "{{ .Values.artifactory.persistence.mountPath }}"

helm/artifactory-values.yaml

@@ -23,7 +23,7 @@ artifactory:
           name: artifactory-volume
   customSidecarContainers: |
     - name: "artifactory-fluentd-sidecar"
-      image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.1"
+      image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.2"
       imagePullPolicy: "IfNotPresent"
       volumeMounts:
         - mountPath: "{{ .Values.artifactory.persistence.mountPath }}"

helm/jfrog-platform-values.yaml

@@ -25,7 +25,7 @@ artifactory:
             name: artifactory-volume
     customSidecarContainers: |
       - name: "artifactory-fluentd-sidecar"
-        image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.0"
+        image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.2"
         imagePullPolicy: "IfNotPresent"
         volumeMounts:
           - mountPath: "{{ .Values.artifactory.persistence.mountPath }}"

helm/xray-values.yaml

@@ -33,13 +33,13 @@ common:
         - '-c'
         - >
           mkdir -p {{ .Values.xray.persistence.mountPath }}/etc/fluentd/;
-          curl https://raw.githubusercontent.com/jfrog/log-analytics-prometheus/PTRENG-5914/fluent_metrics.conf.xray -o {{ .Values.xray.persistence.mountPath }}/etc/fluentd/fluentd.conf;
+          curl https://raw.githubusercontent.com/jfrog/log-analytics-prometheus/{{ .Values.global.jfrog.observability.branch }}/fluent_metrics.conf.xray -o {{ .Values.xray.persistence.mountPath }}/etc/fluentd/fluentd.conf;
       volumeMounts:
         - mountPath: "{{ .Values.xray.persistence.mountPath }}"
           name: data-volume
   customSidecarContainers: |
     - name: "xray-platform-fluentd-sidecar"
-      image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.1"
+      image: "releases-pts-observability-fluentd.jfrog.io/fluentd:4.2"
       imagePullPolicy: "IfNotPresent"
       volumeMounts:
         - mountPath: "{{ .Values.xray.persistence.mountPath }}"