LOGIN
START FREE TRIAL

LogSense Blog

See everything, even before it happens.

How to Combine Tracing and Logs for Faster Troubleshooting

Oct 2, 2019 1:00:10 PM |     Przemek Maciołek, PhD
tracing-and-logs-for-faster-troubleshooting2

Tracing is a basic tool in identifying problems with code. If it's misbehaving or performing poorly, the only way to find out why might be to set tracepoints and watch execution paths in detail. A number of application performance monitoring (APM) systems include a powerful tracing capability.

Another valuable source of information in these situations is logs. Every development team uses either something as simple as a console output or one of the many logging libraries, such as log4j, Python logging, Serilog, or Monolog, to report information as the code runs. They contain a huge amount of information. Unfortunately, most tracing solutions don't provide integration with log files. Pulling log information in is hard. And unless combined, you don't get the observability you need. Shouldn't there be a better way?

Tracing and Custom Logging

APM systems generally do support custom logging as a separate logging framework or API. Developers can not only set simple tracepoints but calculate and report custom information. For example, .NET provides the EventLog class to create event sources and log them as part of the debugging and tracing environment.

There are many ways custom logging can aid diagnosis of problems. To name a few:

Dumping relevant variables

Calculating and reporting metrics on the fly, e.g., the number of times a function has been called

Resource status, e.g., available memory

 

This additional information can help identify the causes of issues and lead to a quicker resolution of problems.

 

Verbosity and Visibility

img-logs-blogThe trouble with logs is that they can be overwhelming - simply too much information. Rather, it's too much irrelevant information and too little in a useful form. People can't wade through megabytes of logs to identify the parts they need.

In their raw form, logs have low visibility and observability. The information is there, but it's very hard to find. This has limited their usefulness in diagnosing problems. When the log level is pushed up to DEBUG, the amount of output becomes huge.

Even with text search tools like grep, developers and testers have to spend hours pulling relevant clues out of the logs. The goal is to find not just single data points but patterns. If a problem is intermittent, the trick is to discover the difference between the times it occurs and the times it doesn't. Putting together the log data that explains such situations is a challenge.

Reinventing the Log

This excess of data is a leading reason for resorting to custom trace logs. The hope is that these calls will provide the necessary information without a lot of noise. In any given case, 99.9% or more of a log file has no applicability to the problem at hand.

But custom logging means redoing work which may already be there. Developers who are already familiar with standard logging systems need to adjust to the quirks of APM custom logging. The attempt to simplify the problem adds a new complication.

What's really needed is to make the available information actionable, not to duplicate it. That means limiting the referenced logs to ones that are relevant to the trace and selecting the entries which are the most directly applicable. In addition, actionable information means organizing those entries in a way that lets developers and testers see just what is going on.

This sounds great in theory, but to write custom logs takes time. What if there were a way -- or a solution that automatically paired traces with logs?

Pulling Logs into Traces with LogSense

LogSense makes it possible to get actionable log information into traces. It knows which logs are associated with which traces, so it can focus on the relevant ones. Even if the log data is basically unstructured, LogSense patent-pending parsing capabilities scan the files and discover patterns automatically. What was originally an intimidating dump of data becomes an organized presentation. Developers can gain insights from it to find the underlying cause of incorrect behavior or slow performance.

img-log-component-blog

By itself, tracing tells the person using it "This is what's going wrong." That's important, but what's really needed is, "This is why it's going wrong." The additional insights from analyzed log data bridge the gap, eliminating hours of puzzling over unexpected code behavior.

The Results

When a problem is difficult to solve, it's often because you're looking in the wrong place. Well-designed logs provide information from multiple angles. They can give developers access to clues which they didn't know they needed. Analyzing their patterns reveals systematic occurrences as well as single events. This is especially important with intermittent problems, where the cause doesn't always show up in a given trace.

Extracting actionable information from logs lets developers find the problems in their code more quickly. Whether the issue is slow performance, crashes, or unexplained user errors, it helps to zero in on the problem by eliminating the irrelevant and organizing the relevant. Buggy code becomes reliable working code, and frustrated developers and managers become happy ones.

Try LogSense today with our free trial - and get your traces and logs working together to help faster troubleshoot in your environment.

 

New call-to-action

 

 

 New call-to-action

Topics: Observability

Want more of the LogSense Blog? You got it.
Subscribe to our newsletter.

Comments