@wmfs/smithereens

smithereens

Smash CSV files into more manageable files based on column values

Install

$ npm install smithereens --save

Usage

const smithereens = require('smithereens')

smithereens(
  [
    '/some/input/csv/files/people.csv'

    // people.csv:
    //
    // personNo,firstName,LastName,personType,action
    // 10,"Lisa","Simpson","c","u"
    // 20,"Homer","Simpson","a","u"
    // 30,"Bart","Simpson","c","d"
    // 40,"Marge","Simpson","a","d"
    // 50,"Maggie","Simpson","c","x"
    // 60,"Grampa","Simpson","x","u"
    // 70,"Milhouse","Van Houten","c","u"

  ],
  {

    outputDirRootPath: '/some/output/dir',

    parser: {
      quote: '"',
      delimiter: ',',
      newline: '\n',
      skipFirstLine: true,
      trimWhitespace: true
    },

    dirSplits: [
      {
        columnIndex: 3,
        valueToDirMap: {
          'c': 'children',
          'a': 'adults'
        }
      }
    ],

    fileSplits: {
      columnIndex: 4,
      valueToFileMap: {
        'u': {
          filename: 'changes',
          outputColumns: [
            {name: 'person_no', columnIndex: 0},
            {name: 'first_name', columnIndex: 1},
            {name: 'last_name', columnIndex: 2}
          ]
        },
        'd': {
          filename: 'deletes',
          outputColumns: [
            {name: 'person_no', columnIndex: 0}
          ]
        }
      }
    }
  },

  function (err, manifest) {

    // File output
    // -----------
    //   /some/output/dir
    //   ./adults
    //     changes.csv:
    //       person_no,first_name,last_name
    //       20,Homer,Simpson
    //     deletes.csv:
    //       person_no
    //       40
    //   ./children
    //     changes.csv:
    //       person_no,first_name,last_name
    //       10,Lisa,Simpson
    //       70,Milhouse,Van Houten
    //     deletes.csv:
    //       person_no
    //       30
    //     unknown.csv:
    //       50,Maggie,Simpson,c,x
    //   ./unknown
    //     changes.csv:
    //       person_no,first_name,last_name
    //       60,Grampa,Simpson

  }
)

smithereens(sourceFilePaths, options, callback)

Arg	Type	Description
sourceFilePaths	string | [string]	A string or an array of strings identifying one or more files. Uses glob so /some/dir/*.csv style patterns are supported, as is directory recursion via /some/dir/**/*.csv
options	object	An object configuring how output should be produced. See Options for more information.
callback	function	To be of the form function(err, manifest). Manifest contains a summary of the output files produced.

Options

Property	Type	Description
outputDirRootPath	string	An absolute directory path where to write output to. All missing directories will be created.
parser	object	An parser object for configuring how input CSV files should be parsed.
dirSplits	[object]	An array of of dirSplit objects
fileSplits	object	A fileSplit object

parser object

Configures how to parse incoming CSV lines. Uses csv-streamify under the bonnet.

Property	Type	Description
skipFirstLine	boolean	Should the first line of each file be ignored? Set to true if files include a header line, for example.
delimiter	string	Comma, semicolon, whatever - defaults to comma.
newline	string	Newline character (use \r\n for CRLF files).
quote	string	What's considered a quote.
empty	string	Empty fields are replaced by this value.

dirSplit object

Smithereens can break CSV files across a nested set of directories based on values defined in each line.

Property	Type	Description
columnIndex	integer	Each line of each CSV file will be parsed into an array of strings. This value identifies which value to split on.
valueToDirMap	object	A simple mapping of an expected string value (as identified by columnIndex) and the directory name that this line should be routed to.

fileSplit object

In a similar way, Smithereens can route lines to different files, based on the contents of a parsed CSV column.

Property	Type	Description
columnIndex	integer	Identifies which of the parsed string values from each CSV line should be used to determine a filename that a row should be routed to.
valueToFileMap	object	A key/value map where key is a string value that is expected via columnIndex and value is a file object.

file object

Defines which filename a CSV row should be routed to, along with some output-formatting configuration.

Property	Type	Description
filename	string	The filename which a row should be routed to. All output files will be in CSV format. Note that the .csv extension is added automatically, so don't include it here.
outputColumns	[object]	An array of outputColumn objects - each defining a column that should appear in the output file.

outputColumn object

Defines the values for each column in the output.

Property	Type	Description
name	string	The name of the column, used in the first line of the CSV output
columnIndex	integer	Identifies the column in the corresponding row of the parsed incoming CSV array to copy to the output. If columnIndex is given any type parameter is ignored.
type	string	Describes other output - hash, uuid, constant. hash generates a hash value from the contents of the corresponding input row, uuid outputs a unique id, constant output a fix value, given by the value property.
value	number or string	A fixed value to put output when the type property is constant.

Testing

$ npm test

License

MIT