asset-smasher

Asset Smasher

Asset pre-processor, merger, and compressor for Node.js

• Structuring Your Assets
  • Manifest Files
  • Manifest Directories
• Using via Command Line
  • Helpers
  • Plugins
• Using via Express Middleware
• Using via Programmatic Interface
• Transformer Notes
  • AMD Support
  • LESS/Stylus
  • ejs
  • dust and Handlebars

Overview

Asset Smasher is a command-line tool, express middleware, and programmatic interface for:

• Pre-processing and transforming files down to plain JavaScript and CSS.
  • .coffee - Compile CoffeeScript into JavaScript
  • .ejs - Run a file through EJS (e.g. to populate configuration parameters into a JavaScript file)
  • .less - Compile Less into CSS
  • .styl - Compile Stylus into CSS
  • .hbs - Precompile Handlebars templates into JavaScript files that register them with Handlebars.templates.
  • .dust - Precompile Dust templates into JavaScript files that register them for use with dust.render.
  • .jsx - Transform JSX files (for use with React) into JavaScript files.
  • Can wrap CommonJS-style JavaScript files with AMD define calls.
  • Processors can be chained together. E.g test.js.hbs.ejs (run Handlebars template through EJS, then compile it)
  • Additional processors can be plugged in.
• Merging files together using Manifest files (.mf) with dependency management directives similar to Sprockets.
  • require - Require a single file
  • require_dir - Require all the files in a specific directory
  • require_tree - Require all the files in a specific directory (and subdirectories)
• Compressing, gzipping, and generating hashed file names.
  • Compress JavaScript files with uglify-js
  • Compress CSS files with ycssmin
  • Generate Gzipped versions of files
  • Include a MD5 hash of the file's contents in the file name. myAsset.js -> myAsset-c89cba7b7df028e65cb01d86f4d27077.js
    • asset_path helper that can be used to reference the hashed name.

It's released under the MIT license.

Structuring Your Assets

Asset Smasher has the concept of "asset paths". These are locations in which your asset files will be located, and from which any relative asset paths will be rooted to.

The simplest structure has one asset path.

E.g.

Asset Paths
-----------
 - app

File Structure
--------------
app/
  js/
  css/
  images/

A more complicated structure might be

Asset Paths
-----------
 - app
 - lib
 - vendor

File Structure
--------------
app/
  js/
  css/
  images/
lib/
  js/
  css/
  images/
vendor/
  js/
  css/
  images/

Both of these examples will result in a compiled structure of

js/
css/
images/

Manifest Files

Manifest (.mf) files are used to merge many assets into a single resulting file. The file should be named with the resulting file type before the .mf extension (e.g. manifest.css.mf or manifest.js.mf. Manifest files can require other manifest files

Only files that will be transformed down to a file of a manifest's "type" (e.g. manifest.css.mf => .css, manifest.js.mf => .js) will be included. This means that, for example, if you require_dir a directory in a JavaScript manifest that happens to contain both JavaScript and CSS, only the JavaScript files will be required.

A simple manifest file might look like

# A comment here
require "./one.js"
require_dir "./subdir1"
#
# Another comment
require_tree "./subdir2"

Directives:

Directive	Description
require "[path]"	Include a single file If the path starts with "/", "../", or "./", process and include the specified file. The file must be inside one of the configured asset paths. If the path does not start with "/", "../", or "./", the file will be searched for in all of the configured asset paths. E.g. if there are asset paths one and two defined, require "js/test.js" will look for one/js/test.js and then two/js/test.js stopping when it finds a matching file. The filename part of the path does not have to include the whole extension. E.g require "test" finds the first file that matches the name in the asset paths (for example test.js.ejs) If the file does not exist/can't be resolved/isn't of the right type for the manifest, it will be ignored (will be logged in verbose mode).
require_dir "[path]"	Include all the files in a directory The path must be absolute, or relative to the current directory. E.g. you can do require_dir "../some/other/dir" but not require_dir "somedir" If using absolute paths, or ".." in your paths, the resulting directory needs to be inside one of the configured asset paths. If the directory does not exist, it will be ignored (will be logged in verbose mode).
require_tree "[path]"	Include the files in a directory recursively The rules for require_tree are the same as the rules for require_dir

Manifest Directories

If you create a directory, for example named foo.js.mf and put a bunch of javascript files in it (or any subdirectories under it), asset-smasher will (recursively) take all the files inside and merge them into foo.js.

Essentially, this is a time-saver so that you don't have to create a manifest file that only contains a single require_tree directive.

Using via Command-Line

Use npm install -g asset-smasher to install the asset-smasher command-line tool globally.

  asset-smasher --help

    Usage: asset-smasher [options] <output dir>

    Options:

      -h, --help               output usage information
      -V, --version            output the version number
      --compress               compress/minify the generated files
      --hash                   generate versions of the files with md5 hashes in the name
      --gzip                   generate gzipped versions of the compiled files
      --hashVersion <version>  invalidate all assets without changing file contents [1.0]
      --only <pattern,...>     only process the files matching these glob patterns (relative to any of the paths) [**/*]
      --paths <path,...>       list of paths to look for assets [.]
      --prefix <prefix>        prefix to append to logical paths when constructing urls. use if output dir is not served from the root of your web app []
      --helpers <js_file>      a .js module of helper functions require()s to expose to transforms []
      --plugins <js_file>      a .js plugin module []
      --verbose                output more verbose information about what is going on to the console
      --noclean                do not delete the output directory before generating files (by default it will be removed first)

    If --only is not specified, *all* files in the --paths will be processed.
    If --hash is specified, a map.json file will be generated that maps the unmangled file name to the hashed one.

    Be careful that your shell doesn't expand glob patterns when passing them as arguments. To be safe, surround the argument with quotes.

    Examples:

      Compile all assets in the current directory to /home/me/compiledAssets

        $ asset-smasher /home/me/compiledAssets

      Something similar to what the Rails asset pipeline does by default

        $ asset-smasher --compress --hash --gzip --prefix /assets \
            --paths ./js,./css,./images \
            --only "**/*.jpg,**/*.gif,**/*.png,application.js.mf,application.css.mf" ./public/assets

      Compile assets, providing some custom helpers to the transformation

        $ asset-smasher --helpers helpers.js output

Helpers

There is a built-in asset_path helper that can be used to get the "real" (i.e. with hashed file name) path of an asset. E.g. asset_path('css/myFile.css') might return '/assets/css/myFile-c89cba7b7df028e65cb01d86f4d27077.css.

Some transformers (e.g. the .ejs one) take in a set of local variables that they can use during transformation. You can pass in the path to a JavaScript module whose exports will be included in this set of variables.

You can use this, for example, to set configuration parameters in your JS files:

helper.js

exports.serviceUrl = 'http://my.service/';

config.js.ejs

//...
var serviceUrl = '<%= serviceUrl %>';
var cssLocation = '<%= asset_path('css/myFile.css') %>';
//...

Execution

$ asset-smasher --helpers helper.js --only config.js.ejs,css/myFile.css .
$ cat config.js
var serviceUrl = 'http://my.service/';
var cssLocation = '/assets/css/myFile-c89cba7b7df028e65cb01d86f4d27077.css';

Plugins

If there's a type of file you want to pre-process that is not natively supported by Asset Smasher, you can add it using a plugin file.

For an example of what the transformer classes look like, look in the lib/compilation/transforms directory

If a plugin module is passed (via --plugins), it will be require()d and then invoked, being passed in the asset smasher library (the module defined in lib/asset-smasher.js)

To register your transformer, just add another entry to the transforms object.

E.g.

my_plugin.js

module.exports = function(assetSmasher) {
   // A stupid transformer that adds "foo" to the start and end of the contents
   var FooTransform = function FooTransform(options) {
     this.options = options || {};
   };
   FooTransform.prototype = {
     extensions:function () {
       return ['.foo'];
     },
     shouldTransform:function (file) {
       return path.extname(file) === '.foo';
     },
     transformedFileName:function (file) {
       return path.basename(file, '.foo');
     },
     transform:function (asset, cb) {
       // Transform the file name
       asset.logicalName = this.transformedFileName(asset.logicalName);
       // Get the contents
       var contents = asset.contents;
       if (Buffer.isBuffer(contents)) {
         contents = contents.toString('utf-8');
       }
       // Compile the contents
       asset.contents = 'foo-' + contents + '-foo';
       cb();
     }
   };

   assetSmasher.transforms.Foo = FooTransform;
};

If you then invoke asset-smasher with --plugins my_plugin.js it will automatically transform *.foo files.

Using via Express Middleware

Asset smasher exposes an express middleware that can:

• Serve your assets un-merged/mangled in development mode.
• Serve precompiled assets (with hashed file names) in production mode.

The middleware takes in the same arguments as the Smasher constructor, with a few extras:

• serve - boolean (or object - see more below) indiacating whether the middleware should serve the asset files. Usualy set this to true in development, false in production (e.g if you're using precompiled assets)
  • An object can be passed in with the following properties to control the serving behavior. If true is passed in, the default values here will be used
    • individual - Whether the individual (true) or merged files (false) should be served. Default true.
• assetMapLocation - path to the map.json generated by the command-line asset-smasher util. This allows the helper methods to determine what the hashed file names were

The middleware exposes two helpers to your views:

• js_asset(logicalPath) - Render a <script> tag for the specified JS asset. When serve is true, this will "explode" manifests and write out a separate <script> for each required file. This makes debugging much easier.
• css_asset(logicalPath) - Render a <link> tag for the specified CSS asset. Same thing happens when serve is true as with js_asset.
• raw_asset(logicalPath) - Return the path to the asset.

You must include the middleware before the Express routing middleware. Otherwise the asset helper functions will not be available for your view to use.

Example

var assetSmasher = require('asset-smasher');

Middleware config (Dev)

app.use(assetSmasher.middleware({
  serve: true,
  paths: [path.join(__dirname, 'assetDir1'), path.join(__dirname, 'assetDir2')],
  prefix: '/assets',
  outputTo: path.join(__dirname, 'tmp')
}));

Middleware config (Prod)

app.use(assetSmasher.middleware({
  serve: false,
  prefix: '/assets',
  assetMapLocation: path.join(__dirname, 'public/assets/map.json')
}));

Middleware config (Alternate Prod config not using precompilation, but instead compile on first access)

Note that if you use this configuration, you will not be able to use "hashed" filenames.

app.use(express.staticCache());
app.use(express.static(path.join(__dirname, 'public')));
app.use(assetSmasher.middleware({
  serve: {
    individual: false
  },
  paths: [path.join(__dirname, 'assetDir1'), path.join(__dirname, 'assetDir2')],
  compress: true,
  prefix: '/assets',
  // This will make the files be served once by asset smasher
  // and then by the express "static" middleware thereafter.
  // You can then also use something like "staticCache" to cache the files if you're not
  // using a reverse proxy cache on the public dir
  outputTo: path.join(__dirname, 'public/assets')
}));

View (ejs here, but could be others)

<!DOCTYPE html>
<html>
<head>
  <title>Test</title>
  <%- css_asset('application.css') %>
  <%- js_asset('application.js') %>
</head>
<body>
  This is a test
</body>
</html>

Using via Programmatic Interface

You can invoke Asset Smasher programmatically by requireing it. You can also plug in additional transformers this way.

The Smasher object has the following methods:

• compileAssets(cb) - Find and compile all the assets.
• compileSingleAsset(assetFilePath, cb) - Compile a single asset (assetFilePath is the actual path to the file, not a logical path)
• findAssets(cb) - Find, but don't compile the assets. Good for determining dependency graph without compiling.
• getAssetByLogicalPath(logicalPath) - Get information about an asset by its logical path. Only call this after finding/compiling assets.
• getHashedFileMapping() - When hash is true, this returns a mapping of logical path to "hashed" logical path. This object is what the command-line tool outputs to map.json. Only call this after finding/compiling assets.
• getRequiredLogicalPathsFor(asset) - Get the logical paths of the assets that should be merged into the specified asset (populated for .mf files). Only call this after finding/compiling assets.
• getProcessingOrderLogicalPaths() - Get a list of the order in which assets should be processed in order to satisfy all dependencies. Only call this after finding/compiling assets.
• reset() - Reset the asset metadata.

The Asset object returned by getAssetByLogicalPath has the following properties (and one method):

• logicalPath - The logical path
• hashedPath - If hash is true, the hashed filename path, otherwise the same as logicalPath
• assetFilePath - The full path to the actual source asset
• compiled - Whether the asset has been compiled
• compiledAssetFilePath - The full path to the compiled asset file
• reset() - Set the asset back to its before-compile state (clear out contents, set name back to pre-transform name)

Example

var assetSmasher = require('asset-smasher');
var Smasher = assetSmasher.Smasher;

// Plug in a custom transformer
assetSmasher.transforms['MyAwesomeFormat'] = require('myAwesomeFormatTransformer');

var sm = new Smasher({
  paths:['/path/one', '/path/two'],
  only:['**/*.{jpg,gif,png}', 'application.js.mf', 'application.css.mf'],
  prefix:'/assets',
  compress:true,
  hash:true,
  hashVersion:'1.0',
  gzip:true,
  outputTo:__dirname + '/public/assets',
  helpers:{
   my: 'helper',
   another: 'helper'
  },
  verbose:true,
  noclean:true
});
sm.compileAssets(function(err) {
  if(err) {
    console.log('An error occurred', err);
  } else {
    console.log('Compilation done!');
  }
});

Transformer Notes

To use a transformer you must install the associated node module.

• .coffee - npm install coffee-script
• .dust - npm install dustjs-linkedin
• .ejs - npm install ejs
• .hbs - npm install handlebars
• .jsx - npm install react-tools
• .less - npm install less
• .styl - npm install stylus

AMD Support

Asset-smasher can wrap JavaScript files that follow the CommonJS module format with Asynchronous Module Definition define calls.

To enable the wrapping of a JavaScript file, put the following at the top of it:

/** @amd */

This will cause asset smasher to:

• Parse the contents of the file for any require calls
• Wrap the contents of the file with a define call where the module id is the logical path of the file (minus .js) and the dependencies are anything that was required. require, exports, and module are always dependencies.

Example (say the file is in scripts/foo.js)

/** @amd */
var x = require('x');
var y = require('../y');

exports.foo = function (bar) {
  return x(bar) + y(bar);
};

This will be transformed into:

define('scripts/foo',
       ['require', 'exports', 'module', 'x', '../y'],
       function (require, exports, module, x, y) {
  var x = require('x');
  var y = require('../y');
  
  exports.foo = function (bar) {
    return x(bar) + y(bar);
  };
});

You can then use an AMD loader (like require.js) to load the modules.

LESS/Stylus

• Any @include/@import paths are relative to the path that the file is in.
• Any @include/@imported files will not be processed individually by Asset Smasher (i.e. you can't @include a LESS file that is preprocessed by ejs)

ejs

• Any registered helpers will be exposed as global variables to the ejs transform.
• The built-in asset_paths helper can be used here.

dust and Handlebars

• The name of the template will be the template's "logical path" (minus the asset path it is in), minus the .js.dust or .js.hbs file extension.
  • E.g. /my/templates/test.js.dust's template name will be test (assuming /my/templates is the asset path)

Changelog

0.3.0 (June 1, 2013)

Functional Changes

• The JSX transformer now can transform .js files (without the additional .jsx extension) as long as the /** @jsx React.DOM */ comment is present in the file.
• For .jsx files, the JSX transformer will add the /** @jsx React.DOM */ comment to the file if it is missing.
• CommonJS-style .js files can now be automatically wrapped with AMD define calls if a /** @amd */ comment is present in the file. See the "AMD Support" section of the README for more info.

API Changes (this should only affect you if you've written custom transformers)

• The shouldTransform and transformedFileName methods in transformers now receive the asset object as a second parameter
• The asset object now has a transformInfo property (that is an object) that transformers can use to put information in when they are processing. This can be used, for example, to prevent a file from being processed more than once (see jsx_transform.js for an example)
• Post-transform operations now must be a constructor function rather than just a function(asset, cb). This is to allow options to be passed into the post-transformer (see end_js_semicolon.js or the new amd_wrapper.js for examples)