Login
Start Trial

Monitoring Python with LogSense

Feb 25, 2019 1:54:53 PM |     Przemek Maciołek, PhD
image (24)

In this tutorial, we will show how to collect metrics from a Python application and analyze them using LogSense.

We will use Flask and logsense-logger for this purpose. In order to start, get your customer token ready.

Part 1 - The Sample App

Step 1: Install Pre-requisites

pip install Flask
pip install https://github.com/collectivesense/logsense-logger-python#egg=logsense-logger

Step 2: Create the App


The app responds to /foo/<value> endpoint and generates some output (this output is done in a way to provide some variation in execution time).

 

from flask import Flask
app = Flask(__name__)


class SomeComplexProcess:
    def __init__(self, how_many_times):
        self._how_many_times = how_many_times

    def foo(self, a):
        x = 0
        for i in range(a*self._how_many_times):
            x = x+1
        return x


example = SomeComplexProcess(10000)


@app.route("/")
def hello():
    return "Hello World!"


@app.route("/foo/<a>")
def test_complex_process(a):
    return "Result: {}".format(example.foo(int(a)))

 

Once created, you can run the app by executing:

FLASK_APP=index.py flask run

and then actually calling it, e.g.:

curl http://localhost:5000/foo/10

 

Step 3: Add Metrics

The LogSense Python library provides a simple annotation, @measure_duration() which calculates the real time taken by the method of execution and passes this information on to LogSense. If the method takes input parameters, they can be specified so they are included as metadata.

Let's add this integration now so the code should look like the following:

from logsense.metrics import measure_duration, setup_metrics

from flask import Flask

# Fetch token
from os import getenv
customer_token = getenv('LOGSENSE_CUSTOMER_TOKEN')

# Setup metrics
setup_metrics('index.py', customer_token)


app = Flask(__name__)


class SomeComplexProcess:
    def __init__(self, how_many_times):
        self._how_many_times = how_many_times

    @measure_duration(extracted_params=['a'])
    def foo(self, a):
        x = 0
        for i in range(a*self._how_many_times):
            x = x+1
        return x


example = SomeComplexProcess(10000)


@app.route("/")
def hello():
    return "Hello World!"


@app.route("/foo/<a>")
def test_complex_process(a):
    return "Result: {}".format(example.foo(int(a)))

 

In order to execute, the token should be provided in the environment variable, e.g. 

LOGSENSE_CUSTOMER_TOKEN=<your token> FLASK_APP=index.py flask run

 

Step 4 - Call Endpoints

You could call the endpoints manually to gather some data but it is better to use `benchmark.py` (available at https://github.com/collectivesense/logsense-logger-python/tree/master/example) together with extended source code for this example and a few more.

To run it, simply call:

./benchmark.py

 

Part 2 - Data Analysis

Step 1 - Find the Logs

After logging-in to the application and typing logs.type=metrics, we can filter by the newly created metrics:

 image (8)

Step 2 - Create Custom Charts with Execution Duration

A custom chart will show us the average duration of foo and hello. For that purpose, go to Views and either click or drag-and-drop Custom Charts on the dashboard:

image (9) 

Now, a blank chart shows up:

 image (10)

We have two ways of selecting the data to plot. The first one is the timestamp method where you select the data used for the buckets from the drop-down menu. Simply click on the "add field" next to the buckets, start typing "timestamp" then select logs.timestamp and hit Apply.

 image (11) 

The Timestamp method tells the custom chart that the data will be plotted by Time.

The other way of plotting a custom chart is the Value method where you can drag and drop the desired key ("logs.foo_duration") on the "values" section:

 image (12)

The chart opens as Count by default. To change it, click on the value to show the drop down, select e.g. AVG, and hit Apply:

 image (13)

Then right-click on the value and select "Open AVG(...)".

  image (14)

Now we have a custom chart that shows the average duration of `foo` over time. Let's also introduce MAX aggregation by adding the same value again - either with the drop-down or drag-and-drop feature:

 image (15)

For the sake of completeness, we also can add MIN aggregation and rename the view:

 image (16) 

As a result, the dashboard appears much more impressive now:

image (17)

Step 3 - Investigate the Data

Looking at the actual logs, one can notice that the duration of the execution seems to be correlated with  parameter 'a'. For example:

 image (18)

When "a" equals to 883, the duration is 1.14s, while with 100 the duration is 0.13s and with 268 the duration is 0.3s7 respectively. In order to find out if it is indeed correlated, we can build a scatter plot.

 

To do so, add a new Custom Chart, drag and drop `logs.a` on the buckets section and `logs.foo_duration` on the values section. Then set the aggregation to AVG and change the chart type to Scatter Plot:

 image (19) 

The correlation is clearly visible now:

 image (20)

Step 4 - Add Additional Information

We also can make another chart that shows how many times the method is being called. The simplest way to do so is to right-click on `logs.foo_duration` and select "Open COUNT(...)" from the drop-down menu:

image (21) 

Then change the rows/buckets to timestamp:

 image (22)

and set the chart type to vertical bars:

image (23) 

Step 5 - Final Edits and Save Dashboard

Once the views have been renamed, they should look like the following:

 image (24)

We can save it as a Dashboard by navigating to the overviews panel and hitting "+" icon:

 image (25) image (26)

As a result, you see it on the list of overviews and are able to browse/view any time.  

Hint 1: Each of the steps in the above instructions is updating the url in the web browser, so you can toggle between the edits of the Dashboard with the Back and Forward browser buttons. You can share individual urls with colleagues (assuming they have a LogSense account) so they can open the same view and edit as needed. 

Hint 2: The Timebar selection is a part of the web browser url but is not retained in the saved overview.

 

Interested to see how LogSense can help with analyzing data from your Python applications?  You can try it out today for free!

Start Trial

 

Topics: appdev

Want more of the LogSense Blog? You got it.
Subscribe to our newsletter.

Comments

Stay Connected

Subscribe to our newsletter.