The emergence of distributed systems brought about a level of administration complexity that engineers couldn’t have foreseen—one that has made collecting and presenting traces, metrics, and logs (telemetry data) significantly more difficult than in monolithic systems.
Most notable is how engineers overseeing the wellness of distributed architectures now have to craft APIs and agents for language-agnostic microservices, in addition to seeing to the instrumentation and presentation of telemetry data from every node/component in their stack. The result was a horde of applications and open-source telemetry collection projects that spewed out inconsistent data.
Often, engineers would (unknowingly) neglect crucial observability points in their blind spots, overwhelmed by the new environment. However, in 2019, with the support of the Cloud Native Computing Foundation (CNCF), Openelemetry rose from the amalgamation of two projects: OpenCensus and OpenTracing.
This article aims to clarify and simplify the concept of OpenTelemetry by explaining why it is necessary while also unbuckling misconceptions about what it does.
A case for OpenTelemetry
OpenTelemetry is an effort to standardize the efforts outlined above through the provision of community-maintained gateways, instrumentation for specific languages and frameworks, and plug-in points for (commercial) observability applications to capture telemetry data from the entire breadth of a distributed architecture.
Why is OpenTelemetry important?
Being a standard instrumentation for sending performance data to any backend of your choice, OpenTelemetry is the bedrock for the improvement of not only the infrastructures on which it lays, but also of the cloud computing industry itself.
Implementing OpenTelemetry inherits the following benefits to your performance measuring efforts:
- Standards make for better telemetry data quality
- Developers get free SDKs and APIs
- Feeding telemetry data to your backend becomes an easier endeavor
- Application performance monitoring (APM) services vendors make better tools
- Scaling opportunities are facilitated
Let’s take a closer look at each of these benefits.
OpenTelemetry upholds quality standards
Being standard and community-developed, the OpenTelemetry project attracts expertise your business benefits from (for free). As a result, the diverse perspectives invested in building a single performance data collector achieve a more thorough product than an isolated group of developers could ever create.
The quality of telemetry data from the open-source community is always increasing and covers more backend languages and frameworks with each turn.
Free APIs and SDKs
The provision of APIs and SDKs makes integrating OpenTelemetry into your stack much easier, regardless of whether you’re working on green or brownfields. For greenfields, building your backend around OpenTelemetry SDKs blends the highest standard of performance data pipelines into your legacy code.
Brownfield development scenarios can leverage APIs and agents to plant collection mechanisms into existing legacy code without changing application logic or process flows. This non-intrusive integration and a plain motive to standardize and improve telemetry data are welcoming, hence the vast community and usage track record of OpenTelemetry thus far.
Aims to make life easier for engineers
Monitoring cloud infrastructures (sometimes multi-cloud) with OpenTelemetry removes the strain of trial and error with in-house-made collections of API and agents. The vibrant community that contributes to the OpenTelemetry framework’s expansion extends technical talent into a company using it.
Sometimes the nodes across a microservice infrastructure use different frameworks and programming languages. The various OpenTelemetry components for diverse backends make it so that one doesn’t have to be a full-stack engineer to monitor and support these microservices.
Implementing collection and exports is an automated process. Furnished with libraries pre-coded with all the right calls for you to take advantage of, the speed of implementation and high-quality results are nothing short of impressive. If you prefer maintaining full knowledge of underlying processes, the integration process between OpenTelemetry and your backend is optionally manual as well.
A vendor-agnostic telemetry bus
By design, OpenTelemetry presents endpoints for data collection applications to plug into, process, and present according to preferences set by their target market. This makes it easy for companies to cycle through vendors in search of the best view (value) into their telemetry data.
What this means is that you can also use a variety of vendors at the same time to cover as much ground into performance monitoring and data visibility as you need to maintain perfect systems.
OpenTelemetry as a springboard for scale
With observability concerns out of the way, engineers can safely expand the surface of their applications. The ability to support different backends concurrently makes OpenTelemetry the perfect bedrock on which to scale applications.
What is OpenTelemetry?
After digesting the history and motives of OpenTelemetry, the “what” should now be easier to grasp. Yes, there have been other respectable telemetry-handling projects in the market, but without some standard measure of quality control, building cloud-native applications would be a slow concept. Every developer would have to reinvent the wheel just for their unique stack.
OpenTelemetry Collector can be implemented into the codebase of applications. This is the simplest implementation. Thereafter, telemetry data will stream towards your set endpoints.
A common mistake is to regard OpenTelemetry in the same set as data visualization tools (commercial or otherwise), when, in fact, they only connect to the endpoints at which the OpenTelemetry Collector feeds telemetry data. As a framework, OpenTelemetry blends with application code, whereas consumers of data simply form a layer outside the application’s logic for processing.
At the time of writing this article, hundreds of companies and the CNCF give OpenTelemetry a resounding nod for taking distributed performance data handling in the right direction.
Some of the biggest observability industry players, including AWS, Elastic, and Splunk, openly integrate OpenTelemetry to form distributions that include their endpoints.
OpenTelemetry typical usage case
A good example to show the implementation of the OpenTelemetry framework is through sharing the common steps across all supported backend technologies. These include a growing list:
- .NET
- Ruby
- C++
- PHP
- Rust
- Erlang/Elixir
- Java
- Go
- Swift
- JavaScript
- Python
Assuming you have an active project, and you wish to import the OpenTelemetry framework into your code, the first step is always adding the free-to-use OpenTelemetry libraries to your dependency files. Depending on the language, this could just be ‘requiring’ the specific libraries into your project globally.
For Rust-lang, this can look like this:
With Ruby, your Gemfile will need to be patched with the following gems:
Once you have added the framework, initiate the code. With JavaScript, this can happen through the installation of community-created and managed instrumentations.
Meanwhile, Golang OpenTelemetry requires that you create spans and attributes for traces as part of the manual instrumentation process.
Most backends have both manual and automatic installation options to streamline integration and break any programming language barrier that might hinder progress. Once the integration is out of the way, you can call export packages to connect your data and ‘dump’ traces into your observability tools.
A Python backend can achieve this by simply running:
The export destination depends on your preferences, but you have a wide variety to pick from. These include a simple writable file, Kafka, a custom HTTP resource, Prometheus, and Zipkin. Of course, you can customize these export endpoints to reduce latency issues when you save telemetry to text files, only to read them again.
Why should engineers care about OpenTelemetry?
From the language-agnostic example above, it’s clear that OpenTelemetry is fast becoming “the way” to push logs, traces, metrics, and any desired data points directly from code into processing and visualization tools. As an engineer, there’s no point swimming against the current, risking poor reporting and sub-par performance monitoring.
As more backend languages join the list supported by OpenTelemetry, it’ll soon be close to impossible to develop cloud-native distributed applications in its absence. As a developer, it’s worth jumping on board early and gaining a technical advantage over the rest of the market.
Final words
Telemetry data compiling and exporting was a wild west until OpenTelemetry became the standard in 2019. A child project of OpenTracing and OpenCensus, OpenTelemetry provides the connectors to integrate with backends and push telemetry data to your preferred data observability tools. The implementation process of OpenTelemetry is simple enough for an engineer to place hooks into backends written in languages barely used before. The open-source community supporting the OpenTelemetry framework keeps adding compatibility with new backend technologies with entire libraries and APIs ready to make data collection a breeze.
The astounding support pouring from tech giants like AWS is a clear signal that engineers must rally behind OpenTelemetry as the default framework to integrate monitoring tools with their stack. Then there are the hundreds of companies openly using OpenTelemetry to export performance data in pursuit of industry dominance.
If you're looking for a powerful platform that makes it easy to connect third-party libraries and manage your applications, then get started with Airplane. Airplane is the developer platform for building internal tools. Using Airplane, you can build custom workflows and UIs using SQL queries, REST API calls, Python scripts, and more. Airplane also offers strong defaults, such as job scheduling and permissions setting, that makes it efficient to build internal tools.
To build your first internal workflow or UI using Airplane within minutes, sign up for a free account or book a demo.
