Security News
ESLint is Now Language-Agnostic: Linting JSON, Markdown, and Beyond
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
csv-streamify
Advanced tools
csv-streamify is an npm package that provides a streaming CSV parser. It allows you to parse CSV data in a streaming fashion, which is useful for handling large datasets without loading the entire dataset into memory.
Streaming CSV Parsing
This feature allows you to parse CSV data in a streaming manner. The code sample demonstrates how to read a CSV file and parse it row by row, logging each row to the console.
const fs = require('fs');
const csvStreamify = require('csv-streamify');
const parser = csvStreamify({ objectMode: true });
fs.createReadStream('data.csv')
.pipe(parser)
.on('data', (row) => {
console.log(row);
})
.on('end', () => {
console.log('CSV parsing completed.');
});
Custom Delimiters
This feature allows you to specify custom delimiters for parsing CSV data. The code sample demonstrates how to parse a CSV file with a semicolon (;) delimiter.
const fs = require('fs');
const csvStreamify = require('csv-streamify');
const parser = csvStreamify({ delimiter: ';', objectMode: true });
fs.createReadStream('data.csv')
.pipe(parser)
.on('data', (row) => {
console.log(row);
})
.on('end', () => {
console.log('CSV parsing completed with custom delimiter.');
});
Handling Headers
This feature allows you to handle CSV headers and map them to object keys. The code sample demonstrates how to parse a CSV file and convert each row into an object with keys corresponding to the headers.
const fs = require('fs');
const csvStreamify = require('csv-streamify');
const parser = csvStreamify({ objectMode: true, columns: true });
fs.createReadStream('data.csv')
.pipe(parser)
.on('data', (row) => {
console.log(row);
})
.on('end', () => {
console.log('CSV parsing completed with headers.');
});
csv-parser is another streaming CSV parser for Node.js. It is similar to csv-streamify in that it allows for streaming parsing of CSV data. However, csv-parser is known for its simplicity and ease of use, making it a popular choice for many developers.
fast-csv is a comprehensive CSV parser and formatter for Node.js. It offers both streaming and non-streaming parsing, as well as CSV formatting capabilities. Compared to csv-streamify, fast-csv provides more features and flexibility, but it may be more complex to use.
papaparse is a powerful CSV parser that works in both Node.js and the browser. It supports streaming, but it is more commonly used for its synchronous parsing capabilities. Compared to csv-streamify, papaparse offers a wider range of features, including support for web workers and large file parsing.
Parses csv files. Accepts options. No coffee script, no weird APIs. Just streams. Tested against csv-spectrum and used in production. It is also "fast enough" (around 60,000 rows per second, but that varies with data obviously).
Works in node 0.12
, 4
, 5
and 6
. Might work in node 0.10
, but is not tested in it.
npm install csv-streamify
This module implements a simple node stream.Transform stream.
You can write to it, read from it and use .pipe
as you would expect.
const csv = require('csv-streamify')
const fs = require('fs')
const parser = csv()
// emits each line as a buffer or as a string representing an array of fields
parser.on('data', function (line) {
console.log(line)
})
// now pipe some data into it
fs.createReadStream('/path/to/file.csv').pipe(parser)
The first argument can either be an options object (see below) or a callback function.
Note: If you pass a callback to csv-streamify
it will buffer the parsed data for you and
pass it to the callback when it's done. This behaviour can obviously lead to out of memory errors with very large csv files.
const csv = require('csv-streamify')
const fs = require('fs')
const parser = csv({ objectMode: true }, function (err, result) {
if (err) throw err
// our csv has been parsed succesfully
result.forEach(function (line) { console.log(line) })
})
// now pipe some data into it
fs.createReadStream('/path/to/file.csv').pipe(parser)
You can pass some options to the parser. All of them are optional.
The options are also passed to the underlying transform stream, so you can pass in any standard node core stream options.
{
delimiter: ',', // comma, semicolon, whatever
newline: '\n', // newline character (use \r\n for CRLF files)
quote: '"', // what's considered a quote
empty: '', // empty fields are replaced by this,
// if true, emit arrays instead of stringified arrays or buffers
objectMode: false,
// if set to true, uses first row as keys -> [ { column1: value1, column2: value2 }, ...]
columns: false
}
Also, take a look at iconv-lite (npm install iconv-lite --save
), it provides pure javascript streaming character encoding conversion.
To use on the command line install it globally:
$ npm install csv-streamify -g
This should add the csv-streamify
command to your $PATH
.
Then, you either pipe data into it or give it a filename:
# pipe data in
$ cat some_data.csv | csv-streamify
# pass a filename
$ csv-streamify some_data.csv > output.json
# tell csv-streamify to read from + wait on stdin
$ csv-streamify -
If you would like to contribute either of those just open an issue so we can discuss it further. :)
Nicolas Hery (objectMode)
FAQs
Streaming CSV Parser. Made entirely out of streams.
We found that csv-streamify demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
Security News
Members Hub is conducting large-scale campaigns to artificially boost Discord server metrics, undermining community trust and platform integrity.
Security News
NIST has failed to meet its self-imposed deadline of clearing the NVD's backlog by the end of the fiscal year. Meanwhile, CVE's awaiting analysis have increased by 33% since June.