logagent-js
Advanced tools
Smart Log Parser and Log Shipper written in Node.
Key features:
How does the parser work? The parser detects log formats based on a pattern library (yaml file) and converts it to a JSON Object:
To test patterns or convert logs from text to JSON use the command line tool 'logagent'. It reads from stdin and outputs line delimited JSON (or pretty JSON or YAML) to the console. In addtion it can forward the parsed objects directly to Logsene.
npm i logagent-js --save
Use the Logparser module in your source code
var Logparser = require('logagent-js')
var lp = new Logparser('./patterns.yml')
lp.parseLine('log message', 'source name', function (err, data) {
if(err) {
console.log('line did not match with any pattern')
}
console.log(JSON.stringify(data))
})
Test your patterns:
cat myapp.log | bin/logagent -y -n myapp -f mypatterns.yml
Official Node.js downloads and instructions. E.g. for Debian/Ubuntu:
curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs
npm i -g logagent-js
# ship all your logs to Logsene, parsed with timestamps - output on console in YAML format (-y)
logagent -t LOGSENE_TOKEN -y /var/log/*.log
The default output is line delimited JSON.
Examples:
# Be Evil: parse all logs
# stream logs to Logsene 1-Click ELK stack
logagent -t LOGSENE_TOKEN /var/log/*.log
# Act as syslog server on UDP and forward messages to Logsene
logagent -t LOGSENE_TOKEN -u 1514
# Act as syslog server on UDP and write YAML formated messages to console
logagent -u 1514 -y
Use a glob pattern to build the file list
logagent -t LOGSENE_TOKEN -g "{/var/log/*.log,/opt/myapp/*.log}"
Watch selective log output on console by passing logs via stdin and format in YAML
tail -f /var/log/access.log | logagent -y
tail -f /var/log/system.log | logagent -f my_own_patterns.yml -y
Ship logs to rtail and Logsene to view logs in real-time in rtail and store logs in Logsene
# rtail don't need to be installed, logagent uses the rtail protocol
logagent -t $LOGSENE_TOKEN --rtail-host myrtailserver --rtail-port 9999 /var/log/*.log
Logagent can start the rtail web-server (in-process, saving memory), open browser with http://localhost:8080
# logagent has no dependency to rtail, to keep the package small
sudo npm i rtail -g
logagent -s -t $LOGSENE_TOKEN --rtail-web-port 8080 --rtail-port 9999 /var/log/*.log
And of course you can combine rtail and Logagent in the traditional way, simply connect both via unix pipes. An example with rtail and Logsene storage and charts:
Heroku can forward logs to a Log Drain
heroku drain:add --app HerokuAppName URL
To receive Heroku logs, logagent-js can be deployed to Heroku. It acts as HTTPS log drain.
or use the following commands:
git clone https://github.com/sematext/logagent-js.git
cd logagent-js
heroku login
heroku create
git push heroku master
export LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN
heroku drains:add --app YOUR_HEROKU_MAIN_APPLICATION `heroku info -s | grep web-url | cut -d= -f2`$LOGSENE_TOKEN
Now you can see your logs in Logsene, define Alert-Queries or use Kibana 4 for Dashboards.
In case of high log volume, scale logagent-js on demand using
heroku scale web=3
Logagent detects the init system and installs systemd or upstart service scripts. On Mac OS X it creates a launchd service. Simply run:
npm i logagent-js -g # install logagent package globally
sudo logagent-setup LOGSENE_TOKEN
The setup script generates a configuraton file in /etc/sematext/logagent.conf.
This file includes the CLI parameters for logagent running as service.
The default settings ship all logs from /var/log/**/*.log to Logsene.
Location of service scripts:
Start/stop service:
service logagent stop/startsystemctl stop/start logagentlaunchctl start/stop com.sematext.logagentThe default pattern definition file include already patterns for:
The file format is based on JS-YAML, in short:
Properties:
Example:
# Sensitive data can be replaced with a hashcode (sha1)
# it applies to fields matching the field names by a regular expression
# Note: this function is not optimized (yet) and might take 10-15% of performance
autohash: !!js/regexp /user|password|email|credit_card_number|payment_info/i
# activate GeoIP lookup
geoIP: true
# logagent updates geoip db files automatically
# pls. note write access to this directory is required
maxmindDbDir: /tmp/
patterns:
- # APACHE Web Logs
sourceName: httpd
match:
# Common Log Format
- regex: !!js/regexp /([0-9a-f.:]+)\s+(-|.+?)\s+(-|.+?)\s+\[([0-9]{2}\/[a-z]{3}\/[0-9]{4}\:[0-9]{2}:[0-9]{2}:[0-9]{2}[^\]]*)\] \"(\S+?)\s(\S*?)\s{0,1}(\S+?)\" ([0-9|\-]+) ([0-9|\-]+)/i
type: apache_access_common
fields: [client_ip,remote_id,user,ts,method,path,http_version,status_code,size]
dateFormat: DD/MMM/YYYY:HH:mm:ss ZZ
# lookup geoip info for the field client_ip
geoIP: client_ip
transform: !!js/function >
function (p) {
p.message = p.method + ' ' + p.path
}
The default patterns are here - contributions are welcome.
FAQs
Smart log parser written in Node
The npm package logagent-js receives a total of 9 weekly downloads. As such, logagent-js popularity was classified as not popular.
We found that logagent-js demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Ransomware payment rates hit an all-time low in 2024 as law enforcement crackdowns, stronger defenses, and shifting policies make attacks riskier and less profitable.
Security News
require(esm) backported to Node.js 20, easing the transition to ESM-only packages and reducing complexity for developers as Node 18 nears end-of-life.
Security News
PyPI now supports iOS and Android wheels, making it easier for Python developers to distribute mobile packages.