quaff

quaff

Key features

• 🚚 A data pipeline helper written in Node.js that works similar to Middleman's Data Files collector
• 📦 Point the library at a folder filled with JS, AML (ArchieML), JSON, YAML, CSV and/or TSV files and get a JavaScript object back that reflects the folder's structure and content/exports
• 🤓 Under the hood it uses parse-json (for better JSON error support), js-yaml and d3-dsv to read files efficiently

Installation

npm install quaff --save-dev

quaff requires Node.js 10 or later.

Usage

Assume a folder with this structure.

data/
  mammals/
    cats.json
    dogs.json
    bears.csv
  birds/
    parrots.yml
    story.aml

After require()'ing quaff:

const quaff = require('quaff');

const data = await quaff('./data/');

console.log(data);

And the results...

{
	"mammals": {
		"cats": ["Marty", "Sammy"],
		"dogs": ["Snazzy", "Cally"],
		"bears": [
			{
				"name": "Steve",
				"type": "Polar bear"
			},
			{
				"name": "Angelica",
				"type": "Sun bear"
			}
		]
	},
	"birds": {
		"parrots": {
			"alive": ["Buzz"],
			"dead": ["Moose"]
		},
		"story": {
			"title": "All about birds",
			"prose": [
				{ "type": "text", "value": "Do you know how great birds are?" },
				{ "type": "text", "value": "Come with me on this journey." }
			]
		}
	}
}

Advanced Usage with JavaScript files

One of the biggest features added in quaff 4.0 is the ability to load JavaScript files. But how exactly does that work?

JavaScript files that are consumed by quaff have to follow one simple rule - they must export a default function (or value) at module.exports. All three of these are valid and return the same value:

module.exports = [
	{
		name: 'Pudge',
		instagram: 'https://instagram.com/pudgethecorgi/',
	},
];

module.exports = () => [
	{
		name: 'Pudge',
		instagram: 'https://instagram.com/pudgethecorgi/',
	},
];

module.exports = async () => [
	{
		name: 'Pudge',
		instagram: 'https://instagram.com/pudgethecorgi/',
	},
];

The final example above is the most interesting one - async functions are supported! This means you can write code to hit API endpoints, or do other asynchronous work, and quaff will wait for those to resolve as it prepares the data object it returns.

const fetch = require('node-fetch');

module.exports = async () => {
	const res = await fetch('https://my-cool-api/');
	const data = await res.json();

	// whatever the API returned will be added to the quaff object!
	return data;
};

Don't have a Promise to do async work with? Working with a callback interface? Just wrap it in one!

const apiHelper = require('my-callback-api');

module.exports = () => {
	return new Promise((resolve, reject) => {
		apiHelper('people', (err, data) => {
			if (err) return reject(err);

			// quaff will take it from here!
			resolve(data);
		});
	});
};

License

By Ryan Murphy.

Available under the MIT license.

Changelog

[4.2.0] - 2020-07-16

Added

• quaff will now throw an error when more than one input file attempts to use the same output key. This is caused by having multiple files in a directory with the same name but different extensions.
• When a .yaml or .yml file fails to parse the thrown error will now include the file path.
• We are now testing quaff in Mac OS and Windows thanks to GitHub Actions. Don't expect that'll ever be an issue but good to know.

Changed

• tiny-glob has been replaced with totalist, which makes quaff a little faster at iterating through files.
• All tests have been moved to uvu.