logagent-js

logagent-js

Smart Log Parser and Log Shipper written in Node.

Key features:

• intelligent pattern matching
• pattern library included
• recognition of Date and Number fields
• easy to extend with custom patterns and JS transform functions
• replace sensitive data with SHA-1 hash codes
• Command Line Tool
  • log format converter (e.g. text to JSON or YAML)
  • Syslog Server (UDP)
  • Heroku Log Drain
  • CloudFoundry Log Drain
  • Log shipper for Logsene from
• Node.js module to integrate parsers into Node.js programs
• logagent-js is part of SPM for Docker to parse Container Logs

How does the parser work? The parser detects log formats based on a pattern library (yaml file) and converts it to a JSON Object:

• find matching regex in pattern library
• tag it with the recognized type
• extract fields using regex
• if 'autohash' is enabled, sensitive data is replaced with its sha1 hash code
• parse dates and detect date format (use 'ts' field for date and time combined)
• create ISO timestamp in '@timestamp' field
• transform function to manipulate parsed objects
• unmatched lines end up with timestamp and original line in the message field
• JSON lines are parsed, and scanned for @timestamp and time fields (logstash and bunyan format)
• default patterns for many applications (see below)
• Heroku logs

To test patterns or convert logs from text to JSON use the command line tool 'logagent'. It reads from stdin and outputs line delimited JSON (or pretty JSON or YAML) to the console. In addtion it can forward the parsed objects directly to Logsene.

Pattern definitions

The default pattern definition file include already patterns for:

• MongoDB,
  • MySQL,
  • Nginx,
  • Redis,
  • Elasticsearch
  • Apache
    • Webserver (httpd),
    • Zookeeper,
    • Cassandra,
    • Kafka,
    • HBase HDFS Data Node,
    • HBase Region Server,
    • Hadoop YARN Node Manager,
    • Apache SOLR,
  • various Linux/Mac OS X system log files

The file format is based on JS-YAML, in short:

• • indicates an array
• !js/regexp - indicates a JS regular expression
• !!js/function > - indicates a JS function

Properties:

• patterns - the list of patterns, each pattern starts with "-"
• match: A group of patterns for a specific log source
• regex: a JS regular expression
• fields: the field list of extracted match groups from the regex
• type: the type used in Logsene (Elasticsearch Mapping)
• dateFormat: the format of the special fields 'ts', if the date format matches, a new field @timestamp is generated
• transform: a JS function to manipulate the result of regex and date parsing

Example:

# Sensitive data can be replaced with a hashcode (sha1)
# it applies to fields matching the field names by a regular expression
# Note: this function is not optimized (yet) and might take 10-15% of performance
# autohash: !!js/regexp /user|client_ip|password|email|credit_card_number|payment_info/i

patterns: 
  - # APACHE  Web Logs
  sourceName: httpd
  match: 
    # Common Log Format
    - regex:        !!js/regexp /([0-9a-f.:]+)\s+(-|.+?)\s+(-|.+?)\s+\[([0-9]{2}\/[a-z]{3}\/[0-9]{4}\:[0-9]{2}:[0-9]{2}:[0-9]{2}[^\]]*)\] \"(\S+?)\s(\S*?)\s{0,1}(\S+?)\" ([0-9|\-]+) ([0-9|\-]+)/i
      type: apache_access_common
      fields:       [client_ip,remote_id,user,ts,method,path,http_version,status_code,size]
      dateFormat: DD/MMM/YYYY:HH:mm:ss ZZ
      transform: !!js/function >
        function (p) {
          p.message = p.method + ' ' + p.path
        }

The default patterns are here - contributions are welcome.

Use logagent-js in Node

npm i logagent-js --save

Use the Logparser module in your source code

var Logparser = require('logagent-js')
var lp = new Logparser('./patterns.yml')
lp.parseLine('log message', 'source name', function (err, data) {
    if(err) {
      console.log('line did not match with any pattern')
    }
    console.log(JSON.stringify(data))
})

Test your patterns:

cat some.log | bin/logagent -y -f mypatterns.yml

Installation for the command line tool

Get Node.js (debian/ubuntu)

# Note the new setup script name for Node.js v0.12
curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
# Then install with:
sudo apt-get install -y nodejs

Install logagent-js command line tool

npm i -g logagent-js
# ship all your logs to logsene, parsed, timestamped - displyed on console in YAML format (-y)
logagent -t LOGSENE_TOKEN -y /var/log/*.log

CLI Parameters:

• -f file with pattern definitions
• -y prints parsed messages in YAML format
• -p pretty json output
• -s silent, print only throughput on exit
• -t token Logsene App Token to insert parsed records into Logsene
• -g use a glob pattern to watch log files e.g. -g "{/var/log/.log,/Users/stefan//*.log}"
• -u UDP_PORT starts a syslogd UDP listener on the given port to act as syslogd
• -n name for the source only when stdin is used (e.g. cat zookeeper.log | logagent -n zookeeper), important to make multi-line patterns working on stdin because the status is tracked by the source name.
• --heroku PORT listens for heroku logs (http drain / framed syslog over http)
• --cfhttp PORT listens for CloudFoundry logs (syslog over http)
• list of files, watched by tail-forver starting at end of file to watch

The default output is line delimited JSON.

Examples:

# Be Evil: parse all logs 
# stream logs to Logsene 1-Click ELK stack 
logagent -t LOGSENE_TOKEN /var/log/*.log 
# Act as syslog server on UDP and forward messages to Logsene
logagent -t LOGSENE_TOKEN -u 1514 
# Act as syslog server on UDP and write YAML formated messages to console
logagent -u 1514 -y  

Use a glob pattern to build the file list

logagent -t LOGSENE_TOKEN -g "{/var/log/*.log,/opt/myapp/*.log}" 

Watch selective log output on console by passing logs via stdin and format in YAML

tail -f /var/log/access.log | logagent -y 
tail -f /var/log/system.log | logagent -f my_own_patterns.yml  -y 

Logagent as Heroku log drain

Heroku can forward logs to a Log Drain

heroku drain:add --app HerokuAppName URL 

To receive Heroku logs, logagent-js can be deployed to Heroku. It acts as HTTPS log drain.

1. deploy logagent-js to Heroku

git clone https://github.com/sematext/logagent-js.git
cd logagent-js
heroku login 
heroku create
git push heroku master

1. Add the the log drain.
   The URL format is https://loggerAppName.herokuapps.com/LOGSENE_TOKEN Use following command, using the dynamically given name from "heroku create".

export LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN
heroku drains:add --app YOUR_HEROKU_MAIN_APPLICATION  `heroku info -s | grep web-url | cut -d= -f2`$LOGSENE_TOKEN

Now you can see your logs in Logsene, define Alert-Queries or use Kibana 4 for Dashboards.

1. Scale logagent-js service on Heroku

In case of high log volume, scale logagent-js on demand using

heroku scale web=3

Logagent as Linux service

Upstart script (ubuntu)

Modify this script and place it in /etc/init/logagent.conf

description "Upstart Logagent"

start on (local-filesystems and net-device-up IFACE=eth0)
stop on runlevel [!12345]

respawn

setuid syslog
setgid syslog
env NODE_ENV=production
env LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN
chdir  /var/log

exec /usr/local/bin/logagent -s /var/log/*.log 

Start the service:

sudo service logagent start

Systemd startup / journald log forwarding

Create a service file for the logagent, in /etc/systemd/system/logagent.service Set the Logsene Token in the "ExecStart" command.

[Service]
Restart=always
RestartSec=1s
ExecStart=/bin/sh -c "journalctl -o json -f | logagent -s -t YOUR_LOGSENE_TOKEN"

[Install]
WantedBy=multi-user.target

Start the service

systemctl start logagent

Related packages

• SPM Agent for Docker - collects metrics, events and logs from Docker API and CoreOS. Logagent-js is a component of spm-agent-docker. More Information: Innovative Docker Log Management
• Logsene-CLI - Enables searching Logsene log entries from the command-line.
• SPM Agent for Node.js - collects performance metrics for Node and io.js applications
• Custom Metrics - Custom Metrics for SPM
• Winston-Logsene - Logging for Node.js - Winston transport layer for Logsene

Support

• Twitter: @sematext
• Blog: blog.sematext.com
• Homepage: www.sematext.com