s3-csv-to-json

s3-csv-to-json

Converts AWS S3 files from CSV to JSON lines via stream with support to gzip for both input and output. Ready to be used as a Node.js module, as a Lambda or via CLI.

Currently it downloads the input file from S3 and upload the result back to it. A support to have one of the both ends pointing to a local file will be worked on.

Goals

• Work with files stored on AWS S3
• Handle large files (stream: Input from S3 -> Convert -> Upload to S3)
• Support to gzip for both input and output
• AWS Lambda ready

Installation

Node.js Module

    npm install s3-csv-to-json

Lambda

Clone the repo and install its dependencies without optionals, as aws-sdk is already available on Lambda functions.

    npm install --no-optional

After installing all dependencies, package it as a zip (e.g. zip -r s3-csv-to-json.zip s3-csv-to-json/*).

Lambda config:

• Code entry type: Upload the zip package.
• Runtime: Node.js 8.10+
• Handler: src/aws-lambda.handler

Usage

Using it as a module, a Lambda or via CLI, it just expects an input and a output path from AWS S3.

{
    "input": "https://s3.amazonaws.com/bucket-name/path/to/input.csv.gz",
    "output": "s3://bucket-name/path/to/output.jsonl.gz"
}

As shown above, S3 protocol is accepted as well for both cases (e.g. s3://bucket-name/path/to/input.csv).

Gzip is supported for both input and output. The .gz extension is all that is needed to determine when the input must be decompressed and if the output should be compressed. If there is no .gz extension it will be handled as a text file (UTF-8).

Node.js Module

const s3CsvToJson = require('s3-csv-to-json.js');

// Assuming that the following is within an async function
const response = await s3CsvToJson({
    input: "https://s3.amazonaws.com/bucket-name/path/to/input.csv.gz",
    output: "s3://bucket-name/path/to/output.jsonl.gz"
});

Lambda

Just post a JSON as already shown above:

{
    "input": "https://s3.amazonaws.com/bucket-name/path/to/input.csv.gz",
    "output": "s3://bucket-name/path/to/output.jsonl.gz"
}

CLI

Clone the repo and execute it using the module MagiCLI via npx:

npx magicli ./path-to-s3-csv-to-json-module --input="s3://bucket-name/path/to/input.csv.gz" --output="s3://bucket-name/path/to/output.jsonl.gz"

Example

CSV Input:

ID,NAME
1,John Doe
2,Jane Doe

JSON lines output:

{"ID":"1","NAME":"John Doe"}
{"ID":"2","NAME":"Jane Doe"}