Commoner
Commoner makes it easy to write scripts that flexibly and efficiently
transpile any dialect of JavaScript into a directory structure of
Node-compatible CommonJS module files.
This task is made possible by
- a declarative syntax for defining how module source code should be
found and processed,
- the use of promises to manage an
asynchronous build pipeline, and
- never rebuilding modules that have already been built.
The output files can be required seamlessly by Node, or served by any
static file server, or bundled together using a tool such as
Browserify or
Stitch for delivery to a web
browser.
Commoner was derived from an earlier, more opinionated tool called
Brigade that provided additional
support for packaging modules together into multiple non-overlapping
bundles. Commoner grew out of the realization that many tools already
exist for bundling CommonJS modules, but that fewer tools focus on getting
to that point.
Installation
From NPM:
npm install commoner
From GitHub:
cd path/to/node_modules
git clone git://github.com/benjamn/commoner.git
cd commoner
npm install .
Usage
Here's the output of bin/commonize --help
:
Usage: commonize [options] <source directory> <output directory> id1 [id2 [id3 ...]]
Options:
-h, --help output usage information
-V, --version output the version number
-c, --config [file] JSON configuration file (no file means STDIN)
-w, --watch Continually rebuild
In a single sentence: the commonize
command finds modules with the given
identifiers id1
, id2
, ... in the source directory and places a
processed copy of each module into the output directory, along with
processed copies of all required modules.
Output
Commoner prints various status messages to STDERR
, so that you can see
what it's doing, or figure out why it's not doing what you thought it
would do.
The only information it prints to STDOUT
is a JSON array of module
identifiers, which includes the identifiers passed on the command line and
all their dependencies. This array contains no duplicates.
Internally, each module that Commoner generates has a hash computed from
the module's identifier, source code, and processing steps. Since this
hash can be computed before processing takes place, Commoner is able to
avoid processing a module if it has ever previously processed the same
module in the same way.
If you dig into the contents of the output directory using ls -la
,
you'll see a subdirectory called .module-cache/
, which contains a bunch
of files with names like 9ffc5c853aac07bc106da1dc1b2486903ca688bf.js
.
When Commoner is about to process a module, it checks its hash against the
file names in this directory. If no match is found, processing procedes
and the resulting file is written to .module-cache/
with a new
hash. Once the appropriate hash file is present in .module-cache/
,
Commoner merely creates a hard link between the hash file and a file with
a more meaningful name outside .module-cache/
.
When you pass the --watch
flag to bin/commonize
, Commoner avoids
exiting after the first build and instead watches for changes to
previously read files, printing a new JSON array of module identifiers to
STDOUT
each time rebuilding finishes. Thanks to the caching of processed
modules, the time taken to rebuild is roughly proportional to the number
of modified files.
Customization
The bin/commonize
script is actually quite simple, and you can write
similar scripts yourself. Let's have a look:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
var context = this;
return context.getProvidedP().then(function(idToPath) {
// If a module declares its own identifier using @providesModule
// then that identifier will be a key in the idToPath object.
if (idToPath.hasOwnProperty(id))
return context.readFileP(idToPath[id]);
});
}, function(id) {
// Otherwise assume the identifier maps directly to a filesystem path.
return this.readFileP(id + ".js");
});
The scriptable interface of the commoner
module abstracts away many of
the annoyances of writing a command-line script. In particular, you don't
have to do any parsing of command-line arguments, and you don't have to
worry about installing any dependencies other than commoner
in your
$NODE_PATH
.
What you are responsible for, at a minimum, is telling Commoner how to
find the source of a module given a module identifier, and you do this by
passing callback functions to require("commoner").resolve
. The script
above uses two strategies that will be tried in sequence: first, it calls
the helper function this.getProvidedP
to retrieve an object mapping
identifiers to file paths (more about this below); and, if that doesn't
work, it falls back to interpreting the identifier as a path relative to
the source directory.
Now, you might not care about this.getProvidedP
. It's really just a
proof of concept that Commoner can support modules that declare their own
identifiers using the // @providesModule <identifier>
syntax, and I
included it by default because it doesn't make a difference unless you
decide to use @providesModule
. If you don't like it, you could write an
even simpler script:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
return this.readFileP(id + ".js");
});
The point is, it's entirely up to you to define how module identifiers are
interpreted. In fact, the source you return doesn't even have to be valid
JavaScript. It could be CoffeeScript, or
LESS, or whatever language you prefer to write by
hand. Commoner doesn't care what your source code looks like, because
Commoner allows you to define arbitrary build steps to turn that source
code into plain old CommonJS.
Let's consider the example of using LESS to write dynamic CSS
modules. First, let's apply what we already know to give special meaning
to .less
files:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
if (isLess(id))
return this.readFileP(id);
}, function(id) {
return this.readFileP(id + ".js");
});
function isLess(id) {
return /\.less$/i.test(id);
}
All this really accomplishes is to avoid appending the .js
file
extension to identifiers that already have the .less
extension.
Now we need to make sure the contents of .less
files somehow get
transformed into plain old CommonJS, and for that we need
require("commoner").process
:
require("commoner").resolve(function(id) {
if (isLess(id))
return this.readFileP(id);
}, function(id) {
return this.readFileP(id + ".js");
}).process(function(id, source) {
if (isLess(id))
return compileLessToJs(source);
return source;
});
How should compileLessToJs
be implemented? At a high level, I propose
that we generate a CommonJS module that will append a new <style>
tag to
the <head>
the first time the module is required. This suggests to me
that we need to take the CSS generated by LESS and somehow embed it as a
string in a CommonJS module with a small amount of boilerplate JS.
Here's a first attempt:
function compileLessToJs(less) {
var css = require("less").render(less);
return 'require("css").add(' + JSON.stringify(css) + ");";
}
Implementing a css
module with an appropriate add
method is an
exercise that I will leave to the reader (hint: you may find this
StackOverflow answer useful).
This almost works, but there's one problem: require("less").render
does
not actually return a string! For better or worse, it passes the compiled
CSS to a callback function, which would make our task extremely painful
if Commoner were not deeply committed to supporting asynchronous
processing.
Commoner uses promises for asynchronous control flow, so we need to return
a promise if we can't return a string immediately. The easiest way to make
a promise is to call this.makePromise
in the following style:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
if (isLess(id))
return this.readFileP(id);
}, function(id) {
return this.readFileP(id + ".js");
}).process(function(id, source) {
if (isLess(id)) {
return this.makePromise(function(nodeStyleCallback) {
compileLessToJs(source, nodeStyleCallback);
});
}
return source;
});
function compileLessToJs(less, callback) {
require("less").render(less, function(err, css) {
callback(err, 'require("css").add(' + JSON.stringify(css) + ");")
});
}
And we're done! This example was admittedly pretty involved, but if you
followed it to the end you now have all the knowledge you need to write
source files like sidebar.less
and require them from other modules by
invoking require("sidebar.less")
. (And, by the way, embedding dynamic
CSS modules in your JavaScript turns out to be an excellent idea.)
Configuration
Of course, not all customization requires modifying code. Most of the
time, in fact, configuration has more to do with providing different
dynamic values to the same code.
For that kind of configuration, you don't need to modify your Commoner
script at all, because Commoner scripts accept a flag called --config
that can either specify a JSON file or (if --config
is given without a
file name) read a string of JSON from STDIN
.
Examples:
bin/commonize source/ output/ main --config release.json
bin/commonize source/ output/ main --config debug.json
echo '{"debug":false}' | bin/commonize source/ output/ main --config
echo '{"debug":true}' | bin/commonize source/ output/ main --config /dev/stdin
This configuration object is exposed to the .resolve
and .process
callbacks as this.config
. So, for example, if you wanted to implement
minification as a processing step, you might do it like this:
require("commoner").resolve(function(id) {
return this.readFileP(id + ".js");
}).process(function(id, source) {
if (this.config.debug)
return source;
return minify(source);
});
Perhaps the coolest thing about the configuration object is that Commoner
generates a recursive hash of all its properties and their values which is
then incorporated into every module hash. This means that two modules with
the same identifier and identical source code and processing steps will
have distinct hashes if built using different configuration objects.