Login
Start Trial

Getting Started

LogSense currently supports two widely used protocols for sending logs: syslog and fluentdTo configure either of them, you must first prepare the CUSTOMER_TOKEN, which was sent to you in the registration confirmation e-mail.

The logs can be sent directly from the source to LogSense or via a relay. The preferred solution depends on the use case:

 

Video instruction on how to use LogSense are available at the bottom of this page.

 

 

 

Fluentd protocol

Logback-logsense for Java (and anything JVM-based)

There's an integration available at https://github.com/collectivesense/logback-logsense

Step 1: Add following dependency to your project:

<dependency>
 <groupId>com.logsense</groupId>
 <artifactId>logback-logsense</artifactId>
 <version>1.0</version>
</dependency>

Step 2: Create src/resources/logback.xml file with following contents (filling-in CUSTOMER_TOKEN with the proper value):

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE logback>
<configuration>

    <!-- LogSense appender. Use the correct accessToken value, as provided by the LogSense app -->
    <appender name="LOGSENSE" class="com.logsense.logback.Appender" >
        <remoteHost>logs.logsense.com</remoteHost>
        <csCustomerToken>CUSTOMER_TOKEN</csCustomerToken>
    </appender>

   <!-- This is just a standard STDOUT appender - keep it (and others) if you intend to use those -->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern><![CDATA[%date{HH:mm:ss.SSS} [%thread] %-5level %logger{15}#%line %X{req.requestURI} %msg\n]]></pattern>
        </encoder>
    </appender>

    <root>
        <level value="DEBUG" />
        <appender-ref ref="STDOUT" />
        <appender-ref ref="LOGSENSE" />
    </root>

</configuration>

(Optional) Testing Sending Logs Through Logback-Logsense

You can clone logback-logsense repository and run the example application.

$ git clone https://github.com/collectivesense/logback-logsense
$ cd logback-logsense/example

(edit src/main/resources/logback.xml using your favorite editor
 and replace CUSTOMER_TOKEN with the actual token sent to you)

$ mvn compile 
$ mvn exec:java -Dexec.mainClass=com.logsense.App

You should be observing in console logs like:

$ mvn exec:java -Dexec.mainClass=com.logsense.App
[INFO] Scanning for projects...
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ example ---
13:33:58.366 [com.logsense.App.main()] INFO  c.l.l.Appender#-2  Using 192.168.1.17 as the source IP address
SLF4J: A number (1) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
13:33:58.412 [com.logsense.App.main()] INFO  c.l.Wombat#26  Current temperature: 129 has risen above 60 degrees.
13:33:58.413 [com.logsense.App.main()] INFO  c.l.Wombat#26  Current temperature: 995 has risen above 60 degrees.
13:33:59.417 [com.logsense.App.main()] INFO  c.l.Wombat#26  Current temperature: 103 has risen above 60 degrees.
13:33:59.418 [com.logsense.App.main()] INFO  c.l.Wombat#26  Current temperature: 944 has risen above 60 degrees.
13:33:59.420 [com.logsense.App.main()] ERROR c.l.Wombat#33  Sanitization failed
java.lang.RuntimeException: That is really low temperature: -447
...

If CUSTOMER_TOKEN was set correctly, you should be able to those logs after logging in to app.logsense.com

WINDOWS EVENT LOGS USING TD-AGENT

The suggested solution for sending Windows Event Logs is to use td-agent - a stable distribution of fluentd prepared by Treasure Data. The installation process is described in detail here: Install by MSI On top of td-agent, in_windows_eventlog plugin provides capabilities to read and push the logs. The exact steps are following:

Step 1: Install Td-Agent

Download the .msi file and install the software.

Step 2: Install Windows_eventlog Input Plugin

Open Td-agent Command Prompt (which was installed by td-agent). In prompt, execute following command:

fluent-gem install fluent-plugin-windows-eventlog

Step 3: Prepare The Config

Open C:\opt\td-agent\etc\td-agent\td-agent.conf and add following entries. Please replace CUSTOMER_TOKEN with the token provided to you in the email. The YOUR_IPvalue should be replaced with the IP you want the logs be visible as the source.

<source>
  @type windows_eventlog
  @id windows_eventlog
  channels application,system,security
  tag winevt.raw
<storage>
@type local
persistent false
</storage> </source> <filter {winevt.raw}.**> @type record_transformer <record> # Please fill the actual token! cs_customer_token "CUSTOMER_TOKEN" # Uncomment and enter IP you want the logs to be # associated with as the source # cs_src_ip "YOUR_IP" cs_pattern_key "description" cs_source_name "eventlog" </record> </filter> <match winevt.raw.**> @type forward # primary host transport tls <server> host logs.logsense.com port 32714 </server> flush_interval 10s </match>

Step 4: Test The Setup

Open Command Prompt as Administrator. Navigate to fluentd directory and run it:

> cd C:\opt\td-agent
> fluentd -c etc\td-agent\td-agent.conf

If the application starts without errors, the logs should be sent already to logsense.com and be visible in the logs view (you can put into search box logs.source_name=eventlog to filter for them).

Hit Ctrl-C to stop the application

Step 5: Register Fluentd As A Service And Run It

Using the Administrator console, execute following commands:

> fluentd --reg-winsvc i
> fluentd --reg-winsvc-fluentdopt '-c C:\opt\td-agent\etc\td-agent\td-agent.conf -o C:\opt\td-agent\td-agent.log'

Go to Control Panel -> System and Security -> Administrative Tools -> Services. Double click on Fluentd Windows Service and click Start button.

Generic fluentd forwarder client setup

It is possible to use generic fluentd forward protocol. In such case following properties must be set:

tag: structured
remote host: logs.logsense.com
remote port: 32714
SSL enabled: true

Additional fields:

(required) cs_customer_token -> with the actual customer token
(optional) cs_pattern_key -> name of the field that will be a subject of automated pattern discovery
(optional) cs_src_ip -> IP address that should be assigned to the logs

 

Syslog protocol

Any syslog compatible source can be used, as long the message template includes source IP and CUSTOMER_TOKEN within square brackets. I.e. 

[IP][CUSTOMER_TOKEN] msg....

Syslog-ng

A template for syslog-ng relay setup is provided below. Note it uses $SOURCEIP macro which fills-in the IP address of the incoming source. You can replace it with a specific value (as long as it is a valid IP address) if the original address is being lost during forwarding the messages.

CUSTOMER_TOKEN must be replaced with the token provided to you.

@version: 3.13

@include "scl.conf"

# LogSense configuration section start

source s_log_sense {
   system(); # Check which OS & collect system logs
   internal(); # Collect syslog-ng logs
};

template LogSenseFormat { template("[$SOURCEIP][CUSTOMER_TOKEN]<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} ${MSG}\n");
   template_escape(no);
};

destination d_log_sense_tls {
   tcp("logs.logsense.com" port(32614)
   tls()
   template(LogSenseFormat));
};

destination d_log_sense_tcp {
   tcp("logs.logsense.com" port(32514)
   template(LogSenseFormat));
};

destination d_log_sense_udp {
   udp("logs.logsense.com" port(32514)
   template(LogSenseFormat));
};

log {
   source(s_log_sense);

   # Please uncomment destination you are interested in, e.g.:
   destination(d_log_sense_tls);
   # destination(d_log_sense_tcp);
   # destination(d_log_sense_udp);
};

# Collective Sense configuration section end
@include "/etc/syslog-ng/conf.d/*.conf"

 

(Optional) Testing Syslog-Ng Relay

If syslog-ng relay was set it should be enough to run logger (typically preinstalled on Linux machines) which sends a message to the syslog acting now as a relay. For instance:

$ logger "Syslog-ng relay test"

Testing syslog protocol

The simplest way to check if logs are being sent is simply using nc (netcat) utility that is usually preinstalled on Linux systems.

To send a log simple you can type following command (replacing IP with any IP address and CUSTOMER_TOKEN with the value that was provided to you):

$ echo "[IP][CUSTOMER_TOKEN] Hello LogSense" | nc -w 5 logs.logsense.com 32514

for instance:

$ echo "[10.1.2.3][aaabbbccc-0011-2233-4455-66778899ddee] Hello LogSense" | nc -w 5 logs.logsense.com 32514

That should produce Hello LogSense log for the specified IP address which should be immediately visible in LogSense.

 

Using the application

HOW TO: View Logs



HOW TO: Edit Log Patterns


HOW TO: Create Charts


HOW TO: View Anomalies