Distributed systems are a collection of autonomous services that interact with one another and are each implemented using the most convenient tools for the job.
Commonly, they are developed using entirely different frameworks and even programming languages. In some situations, they are maintained by very different parts of the software development organization. This distribution is great for scalability, but makes it challenging to have complete visibility to the entire development process.
How did we get to OpenTracing
Distributed systems are popular when developing modern systems because of several reasons, including the increased ease of debugging and modifying systems. However, breaking down a large system into individual microservices results in a loss of observability of how the software works in production. In the absence of a distributed tracing system in place, analyzing the workflows of operations becomes strenuous, particularly pinpointing when and by whom a request is received. This is where OpenTracing steps in, offering a reprieve by revealing a means of correlating together all of the related activities even across service partitions. OpenTracing dissolves the service boundaries and exposes tracing in a vendor-agnostic manner.
Initially, the concerns related to code instrumentation were tightly associated with underlying tracing platforms. To eliminate the problems, there was a need to implement interaction via an API around distributed processing. Moreover, it necessitates setting standards. OpenTracing offered a common technique for the sake of coherency, enabling developers to instrument tracing into their code base across the various programming languages and frameworks. Tracing service providers are gradually getting behind OpenTracing as a standardized way to realize distributed tracing. There are several steps involved to successfully use OpenTracing: instrumenting the software and connecting the tracer.
OpenTracing is a global initiative, and LogSense is proud to support and contribute to the project.
Instrumenting the software
To begin, one has to first instrument the system, and by this, we mean create a layer (calling OpenTracing API) that sits between the origin of the data that we want to gain visibility into (such as the microservices frameworks) and feeding it to the tracer. Luckily, some frameworks make the process easier and faster with automatic instrumentation. Modern distributed systems have many moving parts, and to capture the full flow, each piece needs to be traced.
This allows us to monitor the full flow of the request, track the latency and see where there may be bottlenecks. In short, plain OpenTracing is great:
- Offers a standard, vendor neutral framework
- Uses Google Dapper concepts; supports DAG for Span vs. Trace relationship
- Supports multiple programming languages
- Available in open source and commercial tracer implementations
- Propagates traces between systems
At the same time, it does have some limitations. Among them, perhaps the biggest limitation is that the plain version does not reuse existing logs, which must be rewritten as (essentially) attributes of spans. This is addressed by the LogSense version of the tracer. Another limitation is that many of the implementations send data through an immediate agent process, which needs to be installed and configured on the system (this is not the case for LogSense, which is an agentless solution).
The risks involved
Often, someone becomes interested in tracing mechanisms, pushes for the organization to take it up, but not everyone does. As a consequence, there is incomplete data. It becomes hard to show value, the momentum drops off, and eventually, the tracing initiative ends. Another common problem is under-resourcing the project. Without full commitment, it is a lot of work with no value in the end.
Once you have instrumented your code, you can then initialize the tracer to start quantifying traces. A tracer is a tool that will record the spans and publish them somewhere. Tracers generate records describing each span and this information needs to be combined for all spans covering the trace to provide understanding into the flow of the request.
If the LogSense logging connector is being used, the system automatically tags each log with trace and span information, which allows us to associate this data with a particular request (and even a specific part of it). In other words, out of tens of thousands of logs, the ones related to the trace can be filtered out easily with LogSense. This information provides great insights into identifying bottlenecks and troubleshooting them.
LogSense Distributed Tracing Platform
LogSense provides a seamless OpenTracing-compatible tracer implementation. Along with first-class support for all logs including automated pattern discovery, LogSense offers an agentless approach where traces are sent out directly, without a separate process involved. LogSense enables developers to combine tracing with standard logging via context injection, automated pattern discovery, and ML/DL for fast and accurate anomaly detection.
By using LogSense, one gains the capability to monitor IT system performance. But what's more, with clear representation of the trace components and association of logs with them, LogSense provides a powerful tool allowing for very efficient root cause analysis.
Additionally, LogSense's pattern discovery system makes all logs structured ones, which enables them to be used for dashboards or anomaly detection. Creating a chart describing any of their parameters is as simple as one click away.
To start with tracing, simply download the required instrumentation library and add single line as the config. Details can be found in documentation here. If you're ready to get started, you can sign up for a free trial today.