Change to the proper directory for workshop example:
cd ./workshops/workshops/otel/ecs-ec2
1 - Example ECS EC2 setup scripts to bring up and down an ECS cluster are included: ecs-cli-up.sh and ecs-cli-down.sh
2 - Identify Coralogix Distro for OpenTelemetry container version here before starting
3 - REQUIRED! Review all scripts and update variables IN CAPS
4 - Study official documentation here: https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/opentelemetry/ecs-ec2
These two policies should be tuned for your own production security needs when going into production.
-These two policies should be tuned for your own production security needs when going into production.
+These workshops are designed to help learn and practice observability and practice integrations to the Coralogix platform. They include reference implementations that are easy to deploy and understand.
Unless otherwise specified, these are designed to run on an Ubuntu cloud host environment. They are not intended to be run on a personal computer. Specific workshops have sandbox setup guidelines beyond a cloud Ubuntu host.
These workshops are not documentation- Coralogix platform documentation is located here
"},{"location":"#opentelemetry-info","title":"OpenTelemetry Info","text":"OpenTelemetry Overview and Value Proposition
"},{"location":"#workshops","title":"Workshops","text":""},{"location":"#apm-microservices","title":"APM - Microservices","text":"Kubernetes (Python, Java) - Kubernetes OpenTelemetry Collector for metrics/traces/logs - OpenTelemetry tracing instrumentation for containerized Python and Java apps - Prometheus custom metrics collection - Sample \"bad\" deployment and errors
AWS ECS Fargate (Python) - 2x container microservice, OpenTelemetry Collector, Firelens log router - Complete Fargate task and ECS config instructions
AWS ECS EC2 (Python)
"},{"location":"#apm-monolith","title":"APM - Monolith","text":"Monolith Otel Collector & Host based applications - Python / Linux - Node / Linux - .NET / Windows
"},{"location":"#real-user-monitoring-rum","title":"Real User Monitoring (RUM)","text":"OpenTelemetry is a standard organized by the Cloud Native Compute Foundation
The Cloud Native Compute Foundation publishes standards like Kubernetes and others that are the leading standards in the industry.
OpenTelemetry is comprised of two key elements: - A standard format for metrics, logs, and traces as well as standard protocols for shipping them - OpenTelemetry Software \u00a0\u00a0- OpenTelemetry Collector which can run anywhere (host/container/local) and receives, processes, and exports telemetry \u00a0\u00a0- OpenTelemetry Tracing Instrumentation for many languages that instruments your apps to emit traces and works in two ways \u00a0\u00a0\u00a0\u00a01. Automatic Instrumentation with no code changes and instruments a long list of frameworks for each language \u00a0\u00a0\u00a0\u00a02. Manual Instrumentation that is added to your code for fine grain trace span building
"},{"location":"otel/about-opentelemetry/#why-opentelemetry-because-observability-should-use-universal-standards","title":"Why OpenTelemetry? Because Observability Should Use Universal Standards","text":"Observability means measuring a system based on its outputs.
This is contrast to installing proprietary software for monitoring/remediation purposes- if a system can simply emit telemetry natively, software is no longer needed.
In a complex environment consisting of servers, containers, cloud, many application languages, and mixes and variations of all of the above, Observability is required to be able to proactively and/or in real time discover and repair problems however sytems are so diverse that traditional post-problem find/fix monitoring solutions no longer function or are scalable.
Monitoring systems traditionally required negotiation with vendors to supply \"monitoring software\" and \"agents\" that are threaded througout an environment. This means annual negotiations are required to keep a business online because technical debt of vendor relationships has to be cured on a regular basis. As systems grow larger and more distributed and complex, the private vendor lock on monitoring creates a fatal reliability flaw in the tech stack.
No one negotiates for HTTP, SQL, HTTPS, TCP etc every year- so why should anyone negotiative for observability formats and agents? These should be based open standards and included as part of day-to-day system building and operations.
OpenTelemetry formats and tracing instrumentation allow for devops, operations, and development to scale infinitely without worry of vendor lock-in or future risk to an environment due to proprietary software.
"},{"location":"otel/ecs-ec2/","title":"OpenTelemetry for AWS ECS-EC2","text":""},{"location":"otel/ecs-ec2/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - AWS Account - Intermediate to Advanced skill with ECS-EC2 - Sufficient permissions to configure AWS - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/ecs-ec2/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/ecs-ec2\n1 - Example ECS EC2 setup scripts to bring up and down an ECS cluster are included: ecs-cli-up.sh and ecs-cli-down.sh 2 - Identify Coralogix Distro for OpenTelemetry container version here before starting 3 - REQUIRED! Review all scripts and update variables IN CAPS 4 - Study official documentation here: https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/opentelemetry/ecs-ec2
These two policies should be tuned for your own production security needs when going into production.
"},{"location":"otel/ecs-ec2/#step-4-execute-workshop","title":"Step 4 - Execute Workshop","text":"source 1-ecs-deploy-otel.sh\nsource 2-ecs-createstack-autogen-otel.sh\nsource 3-ecs-deletestack-autogen-otel.sh \nsource 4-ecs-deletestack-otel.sh\nThis example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - AWS Account - Intermediate to Advanced skill with ECS-Fargate - Sufficient permissions to configure AWS - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/ecs-fargate/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/ecs-fargate\n1 - Put the following file in an S3 bucket: https://github.com/coralogix/telemetry-shippers/blob/master/logs/fluent-bit/ecs-fargate/base_filters.conf
2 - Put the following in a Systems Manager Parameter Store: https://github.com/coralogix/telemetry-shippers/blob/master/otel-ecs-fargate/config.yaml
3 - Create a role called ecsTaskExecutionRole that contains the AWS managed AmazonECSTaskExecutionRolePolicy and two example policies from this workshop: ecs-policy-s3-access.json ecs-policy-secrets-access.json
These two policies should be tuned for your own production security needs when going into production.
"},{"location":"otel/ecs-fargate/#step-4-prep-register-task","title":"Step 4: Prep / Register Task","text":"Prep example Task: aws-fargate-otel-demo.json
Update all 14 locations replacing contents bewteen < > and removing those brackets but leaving the quotes: adding task execution role, key, domain, S3 ARN, Systems Parameter etc...
Once prepped the task can be registered and made into a service.
"},{"location":"otel/ecs-fargate/#step-5-launch-task","title":"Step 5: Launch Task","text":"Launch according to your standard procedure.
The task creates the following containers: - two microservices (Client/Server) - FireLens log router - Otel collector
The microservice containers create a Python HTTP client and server that will generate spans and logs and send to Coralogix.
"},{"location":"otel/ecs-fargate/#step-6-study-results-in-coralogix-portal","title":"Step 6 - Study results in Coralogix portal","text":"The shell scripts show how a the microservices example.
"},{"location":"otel/logparsing/logparsing/","title":"OpenTelemetry for Log Observability","text":""},{"location":"otel/logparsing/logparsing/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix Acccount - Cloud Linux host (Debian preferred but RPM flavors will work) - Updated versions and sufficient permissions for installing software - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/logparsing/logparsing/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/logparsing/\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Collector config.yaml must be configured with Coralogix Exporter. See the \"Send Data to Coralogix\" section for determining telemetry endpoint and and API key: https://coralogix.com/docs/guide-first-steps-coralogix/
See the config.yaml example in ./workshops/workshops/otel/logparsing/otelcol This file can be updated in /etc/otelcol/contrib/config.yaml - make a backup of the default version first.
Manage the collector using systemctl i.e. sudo systemctl restart otelcol More info is here: https://coralogix.com/docs/guide-first-steps-coralogix/
You can check Collector status with these status URLS: http://localhost:55679/debug/tracez http://localhost:55679/debug/pipelinez For more info: zpages docs
/tmp/cx gunzip Log.log.gz /etc/otelcol/config.yaml receivers:\n filelog:\n include: [ /tmp/cx/LOG.log ]\n/etc/otelcol/config.yaml service: \n pipelines:\n logs:\n receivers: [otlp, filelog]\nsudo systemctl restart otelcol-contrib.service \nsudo systemctl status otelcol-contrib.service \nThe Explore->Logs option now will show the integrated logs.
This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix acccount - Kubernetes cluster that can be used as a sandbox environment. See Appendix below for a suggested Kubernetes sandbox environment with a quick and easy setup. This setup is recommended for this workshop. - Updated versions and sufficient skill and permissions for helm and kubectl and open source software in general - Proper IDE i.e. Visual Studio Code
Easy Coralogix instructions for Complete Observability are here
"},{"location":"otel/microservices-workshop/#step-2-setup","title":"Step 2 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd workshops/workshops/otel/microservices-demo\nDeploy example to your k8s cluster- this will deploy to the default k8s namespace. If you want to change the namespace, edit yaml/deploy-good.yaml There will be three services spun up:
cx-shopping-cart-reqs - a requesting service initiating a transaction cx-payment-gateway-flask - a server that is a bridge for a transaction to a database- returns a transaction ID to the shopping-cart cx-redis - an instance of a redis database used for a transactionsource deploy-all.sh\nDeploys the following as seen from the http://cx-payment-gateway-flask:5000/ root span:
Note: images may vary from actual UI due to frequent updates:
With healthy low latency spans for all services:
"},{"location":"otel/microservices-workshop/#step-5-simulate-cicd-scenarios","title":"Step 5 - Simulate CI/CD scenarios","text":"
Study results in Coralogix portal
Simulate a \"bad\" deployment:
source deploy-reqs-bad.sh\nThis deployment will cause severe sporadic problems in payment-gateway such as 404s, a log key:value 'transaction': 'failed', and latency in the service response along with a drop in transaction volume. You can see the latency spikes here:
Alerts and automation can be built around span latency or Payment Gateway 404 responses.
Roll back the bad deployment for the services to return to normal:
source deploy-reqs-good.sh\nSpan latency will return to normal and Payment Gateway 404 responses will cease.
"},{"location":"otel/microservices-workshop/#step-6-study-how-opentelemetry-tracing-instrumentation-works","title":"Step 6 - Study how OpenTelemetry tracing instrumentation works","text":"OpenTelemetry Instrumentation For Containerized Apps - Dockerfiles for the containers contain OpenTelemetry auto instrumentation for Python and are in /python root level - Study Otel Python Auto Instrumentation - Notice how the Dockerfile adds the automatic instrumentation - And observe how the kickstart script uses the instrumenting command
Kubernetes Deployments for Otel Tracing Instrumentation - .yaml deployment files are in python/yaml and show how environment variables are used to control the instrumentation - study the deploy-good.yaml which shows the environment variables that control OpenTelemetry tracing instrumentation. Focus on how the Kubernetes Downward API use of status.hostIP to tell the instrumentation where to send traces: to the IP adddress of the host node on port 4317
Instrumented Applications and Frameworks - The Python apps that drive this example are in the python/apps dir - Examine the frameworks used so you can see how tracing picks up their execution - Note that the python requests library is used to make http requests, and that OpenTelemetry Python Instrumentation lists requests as an automatically instrumented library
Deploy Java based Inventory service:
source deploy-java.sh\nDeploy Node.js self contained trace generator- makes requests to a URL specified in environment variables:
source deploy-node-autogen.sh\nTo remove all the deployments/services/pods from the example from your k8s cluster (ignore any errors it reports):
source delete-all.sh\nsudo chmod 644 /etc/rancher/k3s/k3s.yaml && export KUBECONFIG=/etc/rancher/k3s/k3s.yaml This script is included in the workshop in the k3s folder: source ./k3s/k3s.sh This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix acccount - Cloud Linux host (Debian is preferred- workshop instructions are for Debian Linux but if RPM is preferred instructions can be modified) - Updated versions and sufficient permissions for installing open source software - Current verson of node and npm installed https://nodejs.org/ - Ensure that the Node version is current (20 or higher). See https://deb.nodesource.com/ for Ubuntu installation instructions - Proper IDE i.e. Visual Studio Code
Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/monolith-workshop/node/\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Collector config.yaml must be configured with Coralogix Exporter. See the \"Send Data to Coralogix\" section for determining telemetry endpoint and and API key: https://coralogix.com/docs/guide-first-steps-coralogix/
See the config.yaml example in ./workshops/workshops/otel/monolith-workshop/node/otelcol This file can be updated in /etc/otelcol/contrib/config.yaml - make a backup of the default version first.
Manage the collector using systemctl i.e. sudo systemctl restart otelcol More info is here: https://coralogix.com/docs/guide-first-steps-coralogix/
You can check Collector status with these status URLS: http://localhost:55679/debug/tracez http://localhost:55679/debug/pipelinez For more info: zpages docs
Demo app will make requests of https://api.github.com and require `ctrl-c`` to exit.
in ./workshops/workshops/otel/monolith-workshop/node/
Install node.js OpenTelemetry Instrumentation:
setup-node.sh\nSetup environment variables for OpenTelemetry:
setup-node-env.sh\nStart demo app:
start-node.sh\nStop app with ctrl-c when ready.
The shell scripts show how a monolith is set up and instrumented for OpenTelemetry. Traces can be found in Explore->Tracing Additionally, host metrics are available via Coralogix's metrics observability tools.
This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix acccount - Cloud Linux host (Debian is preferred- workshop instructions are for Debian Linux but if RPM is preferred instructions can be modified) - Current versions and sufficient permissions for installing open source software - Current versons (>3.10) of python and pip installed - Proper IDE i.e. Visual Studio Code
Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/monolith-workshop/python/\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Collector config.yaml must be configured with Coralogix Exporter. See the \"Send Data to Coralogix\" section for determining telemetry endpoint and and API key: https://coralogix.com/docs/guide-first-steps-coralogix/
See the config.yaml example in ./workshops/workshops/otel/monolith-workshop/python/otelcol This file can be updated in /etc/otelcol/contrib/config.yaml - make a backup of the default version first.
Manage the collector using systemctl i.e. sudo systemctl restart otelcol More info is here: https://coralogix.com/docs/guide-first-steps-coralogix/
You can check Collector status with these status URLS: http://localhost:55679/debug/tracez http://localhost:55679/debug/pipelinez For more info: zpages docs
python3 --version to check you have python3 installed. If not follow the python3 installation document for your linux distribution. For ubuntu followpip3 --version to check pip3 is installed. If not follow the pip3 installation documentation for your linux distribution. For ubuntu use command sudo apt-get install python3-pipsudo apt upgrade to upgrde already installed packages. If you receive a message to reboot your server do so before moving on to the next steps.This application make 100 requests to https://reqbin.com/echo/get/json and then exit.
Run following scripts found under ./python:
source setup-python.sh\nsource setup-python-env.sh\nsource start-python.sh\nNote: Running source setup-python.sh would output long list of packages and with messages like Successfully installed opentelemetry-instrumentation-some-package.
Explore >> Logs UI Explore >> Tracing UI Individual Trace Setup Correlation (logs in-context) Logs In-context ```
"},{"location":"otel/monolith-workshop/windows/","title":"OpenTelemetry for .NET / Monolith","text":""},{"location":"otel/monolith-workshop/windows/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Windows Server 2019 or newer - Updated versions and sufficient permissions for downloading and installing software and no restrictions on GitHub domain - .NET Framework 3.5 or greater installed - PowerShell 5.1 or higher - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/monolith-workshop/windows/#about-opentelemetry-for-net","title":"About OpenTelemetry For .NET","text":"You can find the complete instruction for .NET OpenTelemetry Agent Instrumentation is in .NET OpenTelemetry GitHub Repository.
There are three key steps to sending traces from a .NET application on Windows: - Install the OpenTelemetry collector - Install OpenTelemetry .NET instrumentation - Run a sample .NET application to test traces
"},{"location":"otel/monolith-workshop/windows/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Use the following example from this repo a template for the OTEL collector config: /workshops/workshops/otel/monolith-workshop/dotnet-windows/otelcol/config.yaml
In config.yaml adjust the domains and your Coralogix private key shown in the snippet below. Also define your application & subsystem name tags.
References: - Coralogix Endpoints - Coralogix Private Key - Application & Subsystem Names
#config.yaml\n\nexporters:\n logging:\n verbosity: detailed\n coralogix:\n # The Coralogix traces ingress endpoint\n traces:\n endpoint: \"YOURDOMAINHERE i.e. ingress.coralogix.us:443\"\n metrics:\n endpoint: \"YOURDOMAINHERE i.e. ingress.coralogix.us:443\"\n logs:\n endpoint: \"YOURDOMAINHERE i.e. ingress.coralogix.us:443\"\n\n # Your Coralogix private key is sensitive\n private_key: \"YOURKEYHERE\"\n\n...\n\n # Traces, Metrics and Logs emitted by this OpenTelemetry exporter \n # are tagged in Coralogix with the default application and subsystem constants.\n application_name: \"MyBusinessEnvironment\"\n subsystem_name: \"MyBusinessSystem\"\nNext: run the collector using the command
otelcol-contrib.exe --config=config.yaml\nYou can find the complete instructions for .NET OpenTelemetry Agent Instrumentation is in .NET OpenTelemetry GitHub Repository.
Below is a shortcut set of instructions but these may be updated in the official documentation above, so we recommend following the officical set.
"},{"location":"otel/monolith-workshop/windows/#quick-start","title":"Quick start","text":"The official Otel instrumentation guide shows the following: Run the commands below in your PowerShell. Update the \"MyServiceDisplayName\" with the name of your application.
# Download the module\n$module_url = \"https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/latest/download/OpenTelemetry.DotNet.Auto.psm1\"\n$download_path = Join-Path $env:temp \"OpenTelemetry.DotNet.Auto.psm1\"\nInvoke-WebRequest -Uri $module_url -OutFile $download_path -UseBasicParsing\n\n# Import the module to use its functions\nImport-Module $download_path\n\n# Install core files\nInstall-OpenTelemetryCore\n\n# Set up the instrumentation for the current PowerShell session\nRegister-OpenTelemetryForCurrentSession -OTelServiceName \"MyServiceDisplayName\"\nNext, configure the follwing Environment Variables
$env:OTEL_RESOURCE_ATTRIBUTES='service.name=dotnetsvc,application.name=dotnetapp cx.application.name=dotnetappcx,cx.subsystem.name=dotnetsubcx'\n$env:OTEL_EXPORTER_OTLP_TRACES_ENDPOINT='localhost:4317'\n$env:OTEL_EXPORTER_OTLP_TRACES_PROTOCOL='grpc'\nFinally, run your dotnet application with
dotnet run\nYou can download & instrument the example application: ./workshops/workshops/otel/monolith-workshop/dotnet-windows/ExampleApp/ provided in this repository for testing.
The example sends traces of http get requests of a public URL. Use dotnet run to start the application.
You may want to open it first in Visual Studio 2022 or higher to ensure dependencies are installed first and then run from the Visual Studio 2022 console.
"},{"location":"otel/monolith-workshop/windows/#aspnet","title":"ASP.NET","text":"IIS / ASP.NET follows a different instruction path for instrumentation and is shown here: https://opentelemetry.io/docs/instrumentation/net/automatic/
"},{"location":"rum/","title":"Real User Metrics","text":""},{"location":"rum/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Intermediate to Advanced skill front-end development / Javascript - Proper IDE i.e. Visual Studio Code
"},{"location":"rum/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/rum\nBrowser SDK and User Context and Labels to a new file called src/index.js source 1-setup-node.sh\ndist into a single dist/main.js filesource 2-webpack.sh\nsource 3-sourcemap.sh\nsource 4-node.sh\nhttp://localhost:3000 to load the app served by Node. Refresh a few times ctrl-c to stop Express serversource 5-cleanup.sh removes all Node packages and webpack artifacts These workshops are designed to help learn and practice observability and practice integrations to the Coralogix platform. They include reference implementations that are easy to deploy and understand.
Unless otherwise specified, these are designed to run on an Ubuntu cloud host environment. They are not intended to be run on a personal computer. Specific workshops have sandbox setup guidelines beyond a cloud Ubuntu host.
These workshops are not documentation- Coralogix platform documentation is located here
"},{"location":"#opentelemetry-info","title":"OpenTelemetry Info","text":"OpenTelemetry Overview and Value Proposition
"},{"location":"#workshops","title":"Workshops","text":""},{"location":"#apm-microservices","title":"APM - Microservices","text":"Kubernetes (Python, Java) - Kubernetes OpenTelemetry Collector for metrics/traces/logs - OpenTelemetry tracing instrumentation for containerized Python and Java apps - Prometheus custom metrics collection - Sample \"bad\" deployment and errors
AWS ECS Fargate (Python) - 2x container microservice, OpenTelemetry Collector, Firelens log router - Complete Fargate task and ECS config instructions
AWS ECS EC2 (Python)
"},{"location":"#apm-monolith","title":"APM - Monolith","text":"Monolith Otel Collector & Host based applications - Python / Linux - Node / Linux - .NET / Windows
"},{"location":"#real-user-monitoring-rum","title":"Real User Monitoring (RUM)","text":"OpenTelemetry is a standard organized by the Cloud Native Compute Foundation
The Cloud Native Compute Foundation publishes standards like Kubernetes and others that are the leading standards in the industry.
OpenTelemetry is comprised of two key elements: - A standard format for metrics, logs, and traces as well as standard protocols for shipping them - OpenTelemetry Software \u00a0\u00a0- OpenTelemetry Collector which can run anywhere (host/container/local) and receives, processes, and exports telemetry \u00a0\u00a0- OpenTelemetry Tracing Instrumentation for many languages that instruments your apps to emit traces and works in two ways \u00a0\u00a0\u00a0\u00a01. Automatic Instrumentation with no code changes and instruments a long list of frameworks for each language \u00a0\u00a0\u00a0\u00a02. Manual Instrumentation that is added to your code for fine grain trace span building
"},{"location":"otel/about-opentelemetry/#why-opentelemetry-because-observability-should-use-universal-standards","title":"Why OpenTelemetry? Because Observability Should Use Universal Standards","text":"Observability means measuring a system based on its outputs.
This is contrast to installing proprietary software for monitoring/remediation purposes- if a system can simply emit telemetry natively, software is no longer needed.
In a complex environment consisting of servers, containers, cloud, many application languages, and mixes and variations of all of the above, Observability is required to be able to proactively and/or in real time discover and repair problems however sytems are so diverse that traditional post-problem find/fix monitoring solutions no longer function or are scalable.
Monitoring systems traditionally required negotiation with vendors to supply \"monitoring software\" and \"agents\" that are threaded througout an environment. This means annual negotiations are required to keep a business online because technical debt of vendor relationships has to be cured on a regular basis. As systems grow larger and more distributed and complex, the private vendor lock on monitoring creates a fatal reliability flaw in the tech stack.
No one negotiates for HTTP, SQL, HTTPS, TCP etc every year- so why should anyone negotiative for observability formats and agents? These should be based open standards and included as part of day-to-day system building and operations.
OpenTelemetry formats and tracing instrumentation allow for devops, operations, and development to scale infinitely without worry of vendor lock-in or future risk to an environment due to proprietary software.
"},{"location":"otel/ecs-ec2/","title":"OpenTelemetry for AWS ECS-EC2","text":""},{"location":"otel/ecs-ec2/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - AWS Account - Intermediate to Advanced skill with ECS-EC2 - Sufficient permissions to configure AWS - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/ecs-ec2/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/ecs-ec2\n1 - Example ECS EC2 setup scripts to bring up and down an ECS cluster are included: ecs-cli-up.sh and ecs-cli-down.sh 2 - Identify Coralogix Distro for OpenTelemetry container version here before starting 3 - REQUIRED! Review all scripts and update variables IN CAPS 4 - Study official documentation here: https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/opentelemetry/ecs-ec2
These two policies should be tuned for your own production security needs when going into production.
"},{"location":"otel/ecs-ec2/#step-4-execute-workshop","title":"Step 4 - Execute workshop","text":"source 1-ecs-deploy-otel.sh\nsource 2-ecs-createstack-autogen-otel.sh\nsource 3-ecs-deletestack-autogen-otel.sh \nsource 4-ecs-deletestack-otel.sh\nThis example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - AWS Account - Intermediate to Advanced skill with ECS-Fargate - Sufficient permissions to configure AWS - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/ecs-fargate/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/ecs-fargate\n1 - Put the following file in an S3 bucket: https://github.com/coralogix/telemetry-shippers/blob/master/logs/fluent-bit/ecs-fargate/base_filters.conf
2 - Put the following in a Systems Manager Parameter Store: https://github.com/coralogix/telemetry-shippers/blob/master/otel-ecs-fargate/config.yaml
3 - Create a role called ecsTaskExecutionRole that contains the AWS managed AmazonECSTaskExecutionRolePolicy and two example policies from this workshop: ecs-policy-s3-access.json ecs-policy-secrets-access.json
These two policies should be tuned for your own production security needs when going into production.
"},{"location":"otel/ecs-fargate/#step-4-prep-register-task","title":"Step 4: Prep / Register Task","text":"Prep example Task: aws-fargate-otel-demo.json
Update all 14 locations replacing contents bewteen < > and removing those brackets but leaving the quotes: adding task execution role, key, domain, S3 ARN, Systems Parameter etc...
Once prepped the task can be registered and made into a service.
"},{"location":"otel/ecs-fargate/#step-5-launch-task","title":"Step 5: Launch Task","text":"Launch according to your standard procedure.
The task creates the following containers: - two microservices (Client/Server) - FireLens log router - Otel collector
The microservice containers create a Python HTTP client and server that will generate spans and logs and send to Coralogix.
"},{"location":"otel/ecs-fargate/#step-6-study-results-in-coralogix-portal","title":"Step 6 - Study results in Coralogix portal","text":"The shell scripts show how a the microservices example.
"},{"location":"otel/logparsing/logparsing/","title":"OpenTelemetry for Log Observability","text":""},{"location":"otel/logparsing/logparsing/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix Acccount - Cloud Linux host (Debian preferred but RPM flavors will work) - Updated versions and sufficient permissions for installing software - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/logparsing/logparsing/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/logparsing/\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Collector config.yaml must be configured with Coralogix Exporter. See the \"Send Data to Coralogix\" section for determining telemetry endpoint and and API key: https://coralogix.com/docs/guide-first-steps-coralogix/
See the config.yaml example in ./workshops/workshops/otel/logparsing/otelcol This file can be updated in /etc/otelcol/contrib/config.yaml - make a backup of the default version first.
Manage the collector using systemctl i.e. sudo systemctl restart otelcol More info is here: https://coralogix.com/docs/guide-first-steps-coralogix/
You can check Collector status with these status URLS: http://localhost:55679/debug/tracez http://localhost:55679/debug/pipelinez For more info: zpages docs
/tmp/cx gunzip Log.log.gz /etc/otelcol/config.yaml receivers:\n filelog:\n include: [ /tmp/cx/LOG.log ]\n/etc/otelcol/config.yaml service: \n pipelines:\n logs:\n receivers: [otlp, filelog]\nsudo systemctl restart otelcol-contrib.service \nsudo systemctl status otelcol-contrib.service \nThe Explore->Logs option now will show the integrated logs.
This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix acccount - Kubernetes cluster that can be used as a sandbox environment. See Appendix below for a suggested Kubernetes sandbox environment with a quick and easy setup. This setup is recommended for this workshop. - Updated versions and sufficient skill and permissions for helm and kubectl and open source software in general - Proper IDE i.e. Visual Studio Code
Easy Coralogix instructions for Complete Observability are here
"},{"location":"otel/microservices-workshop/#step-2-setup","title":"Step 2 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd workshops/workshops/otel/microservices-demo\nDeploy example to your k8s cluster- this will deploy to the default k8s namespace. If you want to change the namespace, edit yaml/deploy-good.yaml There will be three services spun up:
cx-shopping-cart-reqs - a requesting service initiating a transaction cx-payment-gateway-flask - a server that is a bridge for a transaction to a database- returns a transaction ID to the shopping-cart cx-redis - an instance of a redis database used for a transactionsource deploy-all.sh\nDeploys the following as seen from the http://cx-payment-gateway-flask:5000/ root span:
Note: images may vary from actual UI due to frequent updates:
With healthy low latency spans for all services:
"},{"location":"otel/microservices-workshop/#step-5-simulate-cicd-scenarios","title":"Step 5 - Simulate CI/CD scenarios","text":"
Study results in Coralogix portal
Simulate a \"bad\" deployment:
source deploy-reqs-bad.sh\nThis deployment will cause severe sporadic problems in payment-gateway such as 404s, a log key:value 'transaction': 'failed', and latency in the service response along with a drop in transaction volume. You can see the latency spikes here:
Alerts and automation can be built around span latency or Payment Gateway 404 responses.
Roll back the bad deployment for the services to return to normal:
source deploy-reqs-good.sh\nSpan latency will return to normal and Payment Gateway 404 responses will cease.
"},{"location":"otel/microservices-workshop/#step-6-study-how-opentelemetry-tracing-instrumentation-works","title":"Step 6 - Study how OpenTelemetry tracing instrumentation works","text":"OpenTelemetry Instrumentation For Containerized Apps - Dockerfiles for the containers contain OpenTelemetry auto instrumentation for Python and are in /python root level - Study Otel Python Auto Instrumentation - Notice how the Dockerfile adds the automatic instrumentation - And observe how the kickstart script uses the instrumenting command
Kubernetes Deployments for Otel Tracing Instrumentation - .yaml deployment files are in python/yaml and show how environment variables are used to control the instrumentation - study the deploy-good.yaml which shows the environment variables that control OpenTelemetry tracing instrumentation. Focus on how the Kubernetes Downward API use of status.hostIP to tell the instrumentation where to send traces: to the IP adddress of the host node on port 4317
Instrumented Applications and Frameworks - The Python apps that drive this example are in the python/apps dir - Examine the frameworks used so you can see how tracing picks up their execution - Note that the python requests library is used to make http requests, and that OpenTelemetry Python Instrumentation lists requests as an automatically instrumented library
Deploy Java based Inventory service:
source deploy-java.sh\nDeploy Node.js self contained trace generator- makes requests to a URL specified in environment variables:
source deploy-node-autogen.sh\nTo remove all the deployments/services/pods from the example from your k8s cluster (ignore any errors it reports):
source delete-all.sh\nsudo chmod 644 /etc/rancher/k3s/k3s.yaml && export KUBECONFIG=/etc/rancher/k3s/k3s.yaml This script is included in the workshop in the k3s folder: source ./k3s/k3s.sh This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix acccount - Cloud Linux host (Debian is preferred- workshop instructions are for Debian Linux but if RPM is preferred instructions can be modified) - Updated versions and sufficient permissions for installing open source software - Current verson of node and npm installed https://nodejs.org/ - Ensure that the Node version is current (20 or higher). See https://deb.nodesource.com/ for Ubuntu installation instructions - Proper IDE i.e. Visual Studio Code
Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/monolith-workshop/node/\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Collector config.yaml must be configured with Coralogix Exporter. See the \"Send Data to Coralogix\" section for determining telemetry endpoint and and API key: https://coralogix.com/docs/guide-first-steps-coralogix/
See the config.yaml example in ./workshops/workshops/otel/monolith-workshop/node/otelcol This file can be updated in /etc/otelcol/contrib/config.yaml - make a backup of the default version first.
Manage the collector using systemctl i.e. sudo systemctl restart otelcol More info is here: https://coralogix.com/docs/guide-first-steps-coralogix/
You can check Collector status with these status URLS: http://localhost:55679/debug/tracez http://localhost:55679/debug/pipelinez For more info: zpages docs
Demo app will make requests of https://api.github.com and require `ctrl-c`` to exit.
in ./workshops/workshops/otel/monolith-workshop/node/
Install node.js OpenTelemetry Instrumentation:
setup-node.sh\nSetup environment variables for OpenTelemetry:
setup-node-env.sh\nStart demo app:
start-node.sh\nStop app with ctrl-c when ready.
The shell scripts show how a monolith is set up and instrumented for OpenTelemetry. Traces can be found in Explore->Tracing Additionally, host metrics are available via Coralogix's metrics observability tools.
This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Coralogix acccount - Cloud Linux host (Debian is preferred- workshop instructions are for Debian Linux but if RPM is preferred instructions can be modified) - Current versions and sufficient permissions for installing open source software - Current versons (>3.10) of python and pip installed - Proper IDE i.e. Visual Studio Code
Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/otel/monolith-workshop/python/\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Collector config.yaml must be configured with Coralogix Exporter. See the \"Send Data to Coralogix\" section for determining telemetry endpoint and and API key: https://coralogix.com/docs/guide-first-steps-coralogix/
See the config.yaml example in ./workshops/workshops/otel/monolith-workshop/python/otelcol This file can be updated in /etc/otelcol/contrib/config.yaml - make a backup of the default version first.
Manage the collector using systemctl i.e. sudo systemctl restart otelcol More info is here: https://coralogix.com/docs/guide-first-steps-coralogix/
You can check Collector status with these status URLS: http://localhost:55679/debug/tracez http://localhost:55679/debug/pipelinez For more info: zpages docs
python3 --version to check you have python3 installed. If not follow the python3 installation document for your linux distribution. For ubuntu followpip3 --version to check pip3 is installed. If not follow the pip3 installation documentation for your linux distribution. For ubuntu use command sudo apt-get install python3-pipsudo apt upgrade to upgrde already installed packages. If you receive a message to reboot your server do so before moving on to the next steps.This application make 100 requests to https://reqbin.com/echo/get/json and then exit.
Run following scripts found under ./python:
source setup-python.sh\nsource setup-python-env.sh\nsource start-python.sh\nNote: Running source setup-python.sh would output long list of packages and with messages like Successfully installed opentelemetry-instrumentation-some-package.
Explore >> Logs UI Explore >> Tracing UI Individual Trace Setup Correlation (logs in-context) Logs In-context ```
"},{"location":"otel/monolith-workshop/windows/","title":"OpenTelemetry for .NET / Monolith","text":""},{"location":"otel/monolith-workshop/windows/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Windows Server 2019 or newer - Updated versions and sufficient permissions for downloading and installing software and no restrictions on GitHub domain - .NET Framework 3.5 or greater installed - PowerShell 5.1 or higher - Proper IDE i.e. Visual Studio Code
"},{"location":"otel/monolith-workshop/windows/#about-opentelemetry-for-net","title":"About OpenTelemetry For .NET","text":"You can find the complete instruction for .NET OpenTelemetry Agent Instrumentation is in .NET OpenTelemetry GitHub Repository.
There are three key steps to sending traces from a .NET application on Windows: - Install the OpenTelemetry collector - Install OpenTelemetry .NET instrumentation - Run a sample .NET application to test traces
"},{"location":"otel/monolith-workshop/windows/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nDownload and install latest CONTRIB release version from here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
Use the following example from this repo a template for the OTEL collector config: /workshops/workshops/otel/monolith-workshop/dotnet-windows/otelcol/config.yaml
In config.yaml adjust the domains and your Coralogix private key shown in the snippet below. Also define your application & subsystem name tags.
References: - Coralogix Endpoints - Coralogix Private Key - Application & Subsystem Names
#config.yaml\n\nexporters:\n logging:\n verbosity: detailed\n coralogix:\n # The Coralogix traces ingress endpoint\n traces:\n endpoint: \"YOURDOMAINHERE i.e. ingress.coralogix.us:443\"\n metrics:\n endpoint: \"YOURDOMAINHERE i.e. ingress.coralogix.us:443\"\n logs:\n endpoint: \"YOURDOMAINHERE i.e. ingress.coralogix.us:443\"\n\n # Your Coralogix private key is sensitive\n private_key: \"YOURKEYHERE\"\n\n...\n\n # Traces, Metrics and Logs emitted by this OpenTelemetry exporter \n # are tagged in Coralogix with the default application and subsystem constants.\n application_name: \"MyBusinessEnvironment\"\n subsystem_name: \"MyBusinessSystem\"\nNext: run the collector using the command
otelcol-contrib.exe --config=config.yaml\nYou can find the complete instructions for .NET OpenTelemetry Agent Instrumentation is in .NET OpenTelemetry GitHub Repository.
Below is a shortcut set of instructions but these may be updated in the official documentation above, so we recommend following the officical set.
"},{"location":"otel/monolith-workshop/windows/#quick-start","title":"Quick start","text":"The official Otel instrumentation guide shows the following: Run the commands below in your PowerShell. Update the \"MyServiceDisplayName\" with the name of your application.
# Download the module\n$module_url = \"https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/latest/download/OpenTelemetry.DotNet.Auto.psm1\"\n$download_path = Join-Path $env:temp \"OpenTelemetry.DotNet.Auto.psm1\"\nInvoke-WebRequest -Uri $module_url -OutFile $download_path -UseBasicParsing\n\n# Import the module to use its functions\nImport-Module $download_path\n\n# Install core files\nInstall-OpenTelemetryCore\n\n# Set up the instrumentation for the current PowerShell session\nRegister-OpenTelemetryForCurrentSession -OTelServiceName \"MyServiceDisplayName\"\nNext, configure the follwing Environment Variables
$env:OTEL_RESOURCE_ATTRIBUTES='service.name=dotnetsvc,application.name=dotnetapp cx.application.name=dotnetappcx,cx.subsystem.name=dotnetsubcx'\n$env:OTEL_EXPORTER_OTLP_TRACES_ENDPOINT='localhost:4317'\n$env:OTEL_EXPORTER_OTLP_TRACES_PROTOCOL='grpc'\nFinally, run your dotnet application with
dotnet run\nYou can download & instrument the example application: ./workshops/workshops/otel/monolith-workshop/dotnet-windows/ExampleApp/ provided in this repository for testing.
The example sends traces of http get requests of a public URL. Use dotnet run to start the application.
You may want to open it first in Visual Studio 2022 or higher to ensure dependencies are installed first and then run from the Visual Studio 2022 console.
"},{"location":"otel/monolith-workshop/windows/#aspnet","title":"ASP.NET","text":"IIS / ASP.NET follows a different instruction path for instrumentation and is shown here: https://opentelemetry.io/docs/instrumentation/net/automatic/
"},{"location":"rum/","title":"Real User Metrics","text":""},{"location":"rum/#instructions","title":"Instructions","text":"This example is for basic study only and is not documentation. Full documentation: https://coralogix.com/docs/ Requirements: - Intermediate to Advanced skill front-end development / Javascript - Proper IDE i.e. Visual Studio Code
"},{"location":"rum/#step-1-setup","title":"Step 1 - Setup","text":"Clone repo:
git clone https://github.com/coralogix/workshops\nChange to the proper directory for workshop example:
cd ./workshops/workshops/rum\nBrowser SDK and User Context and Labels to a new file called src/index.js source 1-setup-node.sh\ndist into a single dist/main.js filesource 2-webpack.sh\nsource 3-sourcemap.sh\nsource 4-node.sh\nhttp://localhost:3000 to load the app served by Node. Refresh a few times ctrl-c to stop Express serversource 5-cleanup.sh removes all Node packages and webpack artifacts