@alexpekurovsky/logparser

LogParser

LogParser is converter from file with single JSON object per line to CSV file.

This project is done as exam task for IronSource.

Features:

• Converting single file
• Converting all files in directory
• Converting files in directory matching pattern
• Converting files in directory and subdirectories (recursively) matching pattern
• Multiple files processing simultaneously

Installation & Usage

$ npm i @alexpekurovsky/logparser [-g]
$ logparser FILENAME  # to process single file
$ logparser DIRNAME   # to process all files in dir
$ logparser DIRNAME -p PATTERN  # to process all files in dir matching mattern
$ logparser DIRNAME -p PATTERN -r  # to process all files in dir and subdirs matching pattern

Examples

$ logparser /var/log/IronSourceApp/IS_main.log
$ logparser /var/log/IronSourceApp
$ logparser /var/log/IronSourceApp -p '*.log'
$ logparser /var/log -p 'IS_*.log' -r

Demo

$ ll
-rw-r--r-- 1 root root 6651236352 Jun 28 01:55 big.log
-rw-r--r-- 1 root root 3815110456 Jun 28 02:18 big.log.csv
-rw-r--r-- 1 root root       2822 Jun 27 23:20 light.log
-rw-r--r-- 1 root root   51962784 Jun 27 22:40 Mundo-json-2013-01-03.log
-rw-r--r-- 1 root root          0 Jun 28 12:01 Mundo-json-2013-01-03.log.csv

$ logparser -h
usage: logparser [-h] [-p PATTERN] [-r] path

Argparse example

Positional arguments:
  path                  Filename or directory to work on

Optional arguments:
  -h, --help            Show this help message and exit.
  -p PATTERN, --pattern PATTERN
                        Filemask to match files. See https://en.wikipedia.
                        org/wiki/Glob_(programming)
  -r, --recursive       Specify this option if you want to check
                        subdirectories for files


$ logparser . -p '*.log'
Processing big.log ...
`-> big.log.csv exists
  `-> File to be created big.log.csv exists. Do you want to [Override/Skip] it:
    `-> Please answer exactly O/S: S
  `-> You skipped processing of big.log
Processing light.log ...
Processing Mundo-json-2013-01-03.log ...
`-> Mundo-json-2013-01-03.log.csv exists
  `-> File to be created Mundo-json-2013-01-03.log.csv exists. Do you want to [Override/Skip] it: O
  `-> Truncating output file Mundo-json-2013-01-03.log.csv
Processing light.log completed.
Processing Mundo-json-2013-01-03.log completed.


$ ll -h
-rw-r--r-- 1 root root 6,2G Jun 28 01:55 big.log
-rw-r--r-- 1 root root 3,6G Jun 28 02:18 big.log.csv
-rw-r--r-- 1 root root 2,8K Jun 27 23:20 light.log
-rw-r--r-- 1 root root 1,8K Jun 28 12:02 light.log.csv
-rw-r--r-- 1 root root  50M Jun 27 22:40 Mundo-json-2013-01-03.log
-rw-r--r-- 1 root root  29M Jun 28 12:02 Mundo-json-2013-01-03.log.csv

License

Feel free to use this code, if needed

More info

Updates to code to work more efficient on big files after Job Offer :)