couchdb-bulk2

couchdb-bulk2

This is a little command line tool meant to eat newline-delimited JSON (CouchDB documents) on stdin and POSTing them to the _bulk_docs endpoint of a CouchDB server.

This is a fork from couchdb-bulk, with the following modifications:

• posts 1000 docs per bulk request (instead of only 1) when the input is newline-delimited JSON
• drops supports for anything else than newline-delimited JSON (1 doc per line), assuming that getting to that data format is the job of another tool, such as jq
• drops supports for the module interface, only the CLI mode remains
• modernized code and dependencies

Installation

npm install -g couchdb-bulk2

CLI

couchdb-bulk2 url [file]

The [file] argument is optional, if its missing (or if its '-'), input is expected to be piped via stdin

Example:

cat ./test/fixtures/docs.ndjson | couchdb-bulk2 http://localhost:5984/testdb
// OR
couchdb-bulk2 http://localhost:5984/testdb ./test/fixtures/docs.ndjson

couchdb-bulk2 expects the input to be newline-delimited JSON.

See http://jsonlines.org for more info on this format.

Each line should be a single doc:

{ "_id": "one" }
{ "_id": "two" }
{ "_id": "three" }

This newline-delimited JSON format can easily be obtained from a JSON document containing an array of docs using a tool such as jq

cat view_reponse.json | jq -c '.docs[]' | couchdb-bulk2 http://localhost:5984/testdb

Options

• -l, --batch-length <number>: set the number of documents to be sent in bulk to CouchDB per batch (default: 1000)
• -s, --sleep <milliseconds>: defines the amount of time (in milliseconds) to wait once a batch was sent before sending a new one (default: 0)

See also

• couchdb-bulk2 works great in combinaison with ndjson-apply: see data transformation workflow in the Inventaire project