@whatmelon12/csvtojson
Advanced tools
A tool concentrating on converting csv data to JSON with customised parser supporting. Fork of the project adding row index to colParser.
csvtojson module is a comprehensive nodejs csv parser to convert csv to json or column arrays. It can be used as node.js library / command line tool / or in browser. Below are some features:
Here is a free online csv to json convert service utilizing latest csvtojson module.
csvtojson has released version 2.0.0.
v1, open this pageIt is still able to use v1 with csvtojson@2.0.0
// v1
const csvtojsonV1=require("csvtojson/v1");
// v2
const csvtojsonV2=require("csvtojson");
const csvtojsonV2=require("csvtojson/v2");
npm i --save csvtojson
/** csv file
a,b,c
1,2,3
4,5,6
*/
const csvFilePath='<path to csv file>'
const csv=require('csvtojson')
csv()
.fromFile(csvFilePath)
.then((jsonObj)=>{
console.log(jsonObj);
/**
* [
* {a:"1", b:"2", c:"3"},
* {a:"4", b:"5". c:"6"}
* ]
*/
})
// Async / await usage
const jsonArray=await csv().fromFile(csvFilePath);
/**
csvStr:
1,2,3
4,5,6
7,8,9
*/
const csv=require('csvtojson')
csv({
noheader:true,
output: "csv"
})
.fromString(csvStr)
.then((csvRow)=>{
console.log(csvRow) // => [["1","2","3"], ["4","5","6"], ["7","8","9"]]
})
const request=require('request')
const csv=require('csvtojson')
csv()
.fromStream(request.get('http://mywebsite.com/mycsvfile.csv'))
.subscribe((json)=>{
return new Promise((resolve,reject)=>{
// long operation for each json e.g. transform / write into database.
})
},onError,onComplete);
/**
csvStr:
a,b,c
1,2,3
4,5,6
*/
const csv=require('csvtojson')
csv({output:"line"})
.fromString(csvStr)
.subscribe((csvLine)=>{
// csvLine => "1,2,3" and "4,5,6"
})
const csv=require('csvtojson');
const readStream=require('fs').createReadStream(csvFilePath);
const writeStream=request.put('http://mysite.com/obj.json');
readStream.pipe(csv()).pipe(writeStream);
To find more detailed usage, please see API section
$ npm i -g csvtojson
$ csvtojson [options] <csv file path>
Convert csv file and save result to json file:
$ csvtojson source.csv > converted.json
Pipe in csv data:
$ cat ./source.csv | csvtojson > converted.json
Print Help:
$ csvtojson
require('csvtojson') returns a constructor function which takes 2 arguments:
const csv=require('csvtojson')
const converter=csv(parserParameters, streamOptions)
Both arguments are optional.
For Stream Options please read Stream Option from Node.JS
parserParameters is a JSON object like:
const converter=csv({
noheader:true,
trim:true,
})
Following parameters are supported:
true if version < 1.1.4)headName: <String | Function | ColParser> . e.g. {field1:'number'} will use built-in number parser to convert value of the field1 column to number. For more information See details beloweol like \n) as a row. This will prevent eol characters from being used within a row (even inside a quoted field). Default is false. Change to true if you are confident no inline line breaks (like line break in a cell which has multi line text)..then is called (or await is used). If this is not desired, set this to false. Default is true.
All parameters can be used in Command Line tool.Since v2.0.0, asynchronous processing has been fully supported.
e.g. Process each JSON result asynchronously.
csv().fromFile(csvFile)
.subscribe((json)=>{
return new Promise((resolve,reject)=>{
// Async operation on the json
// don't forget to call resolve and reject
})
})
For more details please read:
Converter class defined a series of events.
header event is emitted for each CSV file once. It passes an array object which contains the names of the header row.
const csv=require('csvtojson')
csv()
.on('header',(header)=>{
//header=> [header1, header2, header3]
})
header is always an array of strings without types.
data event is emitted for each parsed CSV line. It passes buffer of stringified JSON in ndjson format unless objectMode is set true in stream option.
const csv=require('csvtojson')
csv()
.on('data',(data)=>{
//data is a buffer object
const jsonStr= data.toString('utf8')
})
error event is emitted if any errors happened during parsing.
const csv=require('csvtojson')
csv()
.on('error',(err)=>{
console.log(err)
})
Note that if error being emitted, the process will stop as node.js will automatically unpipe() upper-stream and chained down-stream1. This will cause end event never being emitted because end event is only emitted when all data being consumed 2. If need to know when parsing finished, use done event instead of end.
done event is emitted either after parsing successfully finished or any error happens. This indicates the processor has stopped.
const csv=require('csvtojson')
csv()
.on('done',(error)=>{
//do some stuff
})
if any error during parsing, it will be passed in callback.
the hook -- preRawData will be called with csv string passed to parser.
const csv=require('csvtojson')
// synchronouse
csv()
.preRawData((csvRawData)=>{
var newData=csvRawData.replace('some value','another value');
return newData;
})
// asynchronous
csv()
.preRawData((csvRawData)=>{
return new Promise((resolve,reject)=>{
var newData=csvRawData.replace('some value','another value');
resolve(newData);
})
})
The function is called each time a file line has been parsed in csv stream. The lineIdx is the file line number in the file starting with 0.
const csv=require('csvtojson')
// synchronouse
csv()
.preFileLine((fileLineString, lineIdx)=>{
if (lineIdx === 2){
return fileLineString.replace('some value','another value')
}
return fileLineString
})
// asynchronous
csv()
.preFileLine((fileLineString, lineIdx)=>{
return new Promise((resolve,reject)=>{
// async function processing the data.
})
})
To transform result that is sent to downstream, use .subscribe method for each json populated.
const csv=require('csvtojson')
csv()
.subscribe((jsonObj,index)=>{
jsonObj.myNewKey='some value'
// OR asynchronously
return new Promise((resolve,reject)=>{
jsonObj.myNewKey='some value';
resolve();
})
})
.on('data',(jsonObj)=>{
console.log(jsonObj.myNewKey) // some value
});
csvtojson is able to convert csv line to a nested JSON by correctly defining its csv header row. This is default out-of-box feature.
Here is an example. Original CSV:
fieldA.title, fieldA.children.0.name, fieldA.children.0.id,fieldA.children.1.name, fieldA.children.1.employee.0.name,fieldA.children.1.employee.1.name, fieldA.address.0,fieldA.address.1, description
Food Factory, Oscar, 0023, Tikka, Tim, Joe, 3 Lame Road, Grantstown, A fresh new food factory
Kindom Garden, Ceil, 54, Pillow, Amst, Tom, 24 Shaker Street, HelloTown, Awesome castle
The data above contains nested JSON including nested array of JSON objects and plain texts.
Using csvtojson to convert, the result would be like:
[{
"fieldA": {
"title": "Food Factory",
"children": [{
"name": "Oscar",
"id": "0023"
}, {
"name": "Tikka",
"employee": [{
"name": "Tim"
}, {
"name": "Joe"
}]
}],
"address": ["3 Lame Road", "Grantstown"]
},
"description": "A fresh new food factory"
}, {
"fieldA": {
"title": "Kindom Garden",
"children": [{
"name": "Ceil",
"id": "54"
}, {
"name": "Pillow",
"employee": [{
"name": "Amst"
}, {
"name": "Tom"
}]
}],
"address": ["24 Shaker Street", "HelloTown"]
},
"description": "Awesome castle"
}]
In order to not produce nested JSON, simply set flatKeys:true in parameters.
/**
csvStr:
a.b,a.c
1,2
*/
csv({flatKeys:true})
.fromString(csvStr)
.subscribe((jsonObj)=>{
//{"a.b":1,"a.c":2} rather than {"a":{"b":1,"c":2}}
});
csvtojson uses csv header row as generator of JSON keys. However, it does not require the csv source containing a header row. There are 4 ways to define header rows:
headers:[] and noheader:false parameters.headers:[] and noheader:true parameters.noheader:true. This will automatically add fieldN header to csv cells// replace header row (first row) from original source with 'header1, header2'
csv({
noheader: false,
headers: ['header1','header2']
})
// original source has no header row. add 'field1' 'field2' ... 'fieldN' as csv header
csv({
noheader: true
})
// original source has no header row. use 'header1' 'header2' as its header row
csv({
noheader: true,
headers: ['header1','header2']
})
Column Parser allows writing a custom parser for a column in CSV data.
What is Column Parser
When csvtojson walks through csv data, it converts value in a cell to something else. For example, if checkType is true, csvtojson will attempt to find a proper type parser according to the cell value. That is, if cell value is "5", a numberParser will be used and all value under that column will use the numberParser to transform data.
There are currently following built-in parser:
This will override types infered from checkType:true parameter. More built-in parsers will be added as requested in issues page.
Example:
/*csv string
column1,column2
hello,1234
*/
csv({
colParser:{
"column1":"omit",
"column2":"string",
},
checkType:true
})
.fromString(csvString)
.subscribe((jsonObj)=>{
//jsonObj: {column2:"1234"}
})
Sometimes, developers want to define custom parser. It is able to pass a function to specific column in colParser.
Example:
/*csv data
name, birthday
Joe, 1970-01-01
*/
csv({
colParser:{
"birthday":function(item, head, resultRow, row , colIdx){
/*
item - "1970-01-01"
head - "birthday"
resultRow - {name:"Joe"}
row - ["Joe","1970-01-01"]
colIdx - 1
*/
return new Date(item);
}
}
})
Above example will convert birthday column into a js Date object.
The returned value will be used in result JSON object. Returning undefined will not change result JSON object.
It is also able to mark a column as flat:
/*csv string
person.comment,person.number
hello,1234
*/
csv({
colParser:{
"person.number":{
flat:true,
cellParser: "number" // string or a function
}
}
})
.fromString(csvString)
.subscribe((jsonObj)=>{
//jsonObj: {"person.number":1234,"person":{"comment":"hello"}}
})
Very much appreciate any types of donation and support.
csvtojson follows github convention for contributions. Here are some steps:
npm test locally before pushing code back.Thanks all the contributors
Thank you to all our backers! [Become a backer]
Thank you to all our sponsors! (please ask your company to also support this open source project by becoming a sponsor)
To use csvtojson in browser is quite simple. There are two ways:
1. Embed script directly into script tag
There is a pre-built script located in browser/csvtojson.min.js. Simply include that file in a script tag in index.html page:
<script src="node_modules/csvtojson/browser/csvtojson.min.js"></script>
<!-- or use cdn -->
<script src="https://cdn.rawgit.com/Keyang/node-csvtojson/d41f44aa/browser/csvtojson.min.js"></script>
then use a global csv function
<script>
csv({
output: "csv"
})
.fromString("a,b,c\n1,2,3")
.then(function(result){
})
</script>
2. Use webpack or browserify
If a module packager is preferred, just simply require("csvtojson"):
var csv=require("csvtojson");
// or with import
import * as csv from "csvtojson";
//then use csv as normal
FAQs
A tool concentrating on converting csv data to JSON with customised parser supporting. Fork of the project adding row index to colParser.
The npm package @whatmelon12/csvtojson receives a total of 2 weekly downloads. As such, @whatmelon12/csvtojson popularity was classified as not popular.
We found that @whatmelon12/csvtojson demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.