depack

depack

depack Is The Bundler To Create Front-End (JS) Bundles And Back-End (Node.JS) Compiled Packages With Google Closure Compiler.

yarn add -E depack

Table Of Contents

• Table Of Contents
• GCC Installation
• CLI
• Bundle Mode
• Compile Mode
  • Usage
  • Troubleshooting
    • Bugs In GCC
    • External APIs
• API
• Wiki
• Org Structure
• Notes
• Copyright & License

GCC Installation

Depack has been built to contain no dependencies to prove its concept. Google Closure Compiler is not installed by it, because the general use-case is to reuse Depack across many projects, and it does not make sense to download and install GCC in each of them in the node_modules folder. Therefore, the recommended way is to install GCC in the home or projects directory, e.g., /Users/home/user or /Users/home/js-projects. In that way, the single GCC will be accessible from there even when running Depack from a particular project (because Node.js will try to resolve the module by traversing up to the root).

The other way to install GCC is to set the GOOGLE_CLOSURE_COMPILER environment variable to point to the compiler, either downloaded from the internet, or built yourself.

CLI

Depack can be used from the command line interface to create bundles or compiled packages for the given entry file.

depack -h

Google Closure Compiler-based packager for the web and Node.JS.
https://artdecocode.com/depack/
Generic flags: https://github.com/google/closure-compiler/wiki/Flags-and-Options

  depack SOURCE [-cl] [-o output.js] [-IO 2018] [-wVvh] [-lvl LEVEL -a] [... --generic-args]

	source            	The source entry to build.
	--output, -o      	Where to save the output.
	                  	Prints to `stdout` when not passed.
	--debug, -d       	The location of the file where to save sources after
	                  	each pass.
	--pretty-print, -p	Whether to apply the `--formatting=PRETTY_PRINT` flag.
	--no-sourcemap, -S	Disable source maps.
	--verbose, -V     	Print the exact command.
	--language_in, -I 	The language of the input sources, years also accepted.
	--language_out, -O	The language spec of the output, years accepted.
	--level, -lvl     	The compilation level. Options:
	                  	BUNDLE, WHITESPACE_ONLY, SIMPLE (default), ADVANCED.
	--advanced, -a    	Whether to enable advanced optimisation.
	--no-warnings, -w 	Do not print compiler's warnings by adding the
	                  	`--warning_level QUIET` flag.
	--version, -v     	Shows the current _Depack_ and _GCC_ versions.
	--help, -h        	Prints the usage information.

BACKEND: Creates a single executable Node.JS file or a library.
  depack SOURCE -cl [-o output.js] [-s]

	--compile, -c  	Set the _Depack_ mode to "compile" to create a Node.JS binary.
	               	Adds the `#!usr/bin/env node` at the top and sets +x permission.
	--no-strict, -s	Whether to remove the `"use strict"` from the output.

  Example:

    depack src/bin.js -c -a -o depack/bin.js -p

FRONTEND: Creates a bundle for the web.
  depack SOURCE [-o output.js] [-H]

	--iife, -i    	Add the IIFE flag to prevent name clashes.
	--temp        	The path to the temp directory used to transpile JSX files.
	              	Default: depack-temp.
	--preact, -H  	Add the `import { h } from "preact"` to JSX files automatically.
	              	Does not process files found in the `node_modules`, because
	              	they are not placed in the temp, and must be built separately,
	              	e.g., with ÀLaMode transpiler.
	--external, -E	The `preact` dependency in `node_modules` will be temporary
	              	renamed to `_preact`, and a monkey-patching package that
	              	imports `＠externs/preact` will take its place. This is to allow
	              	bundles to import from _Preact_ installed as a script on a webpage,
	              	but exclude it from compilation. `preact` will be restored at the end.
	--patch, -P   	Patches the `preact` directory like in `external`, and waits for
	              	user input to restore it. Useful when linking packages and wanting
	              	to them from other projects.

  Example:

    depack source.js -o bundle.js -i -a -H

Depack supports the following flags for both modes. Any additional arguments that are not recognised, will be passed directly to the compiler. For mode-specific arguments, see the appropriate section in this README.

Argument	Short	Description
source		The source entry to build.
--output	-o	Where to save the output. Prints to stdout when not passed.
--debug	-d	The location of the file where to save sources after each pass.
--pretty-print	-p	Whether to apply the --formatting=PRETTY_PRINT flag.
--no-sourcemap	-S	Disable source maps.
--verbose	-V	Print the exact command.
--language_in	-I	The language of the input sources, years also accepted.
--language_out	-O	The language spec of the output, years accepted.
--level	-lvl	The compilation level. Options: BUNDLE, WHITESPACE_ONLY, SIMPLE (default), ADVANCED.
--advanced	-a	Whether to enable advanced optimisation.
--no-warnings	-w	Do not print compiler's warnings by adding the --warning_level QUIET flag.
--version	-v	Shows the current Depack and GCC versions.
--help	-h	Prints the usage information.

Bundle Mode

Depack comes packed with a JSX transpiler that is based on Regular Expressions transforms. There are some limitations like currently non working comments, or inability to place {} and <> strings and functions (although the arrow functions are supported), but it works. What is also important is that the parser will quote the properties intended for html elements, but leave the properties unquoted for the components.

This means that the properties' names will get mangled by the compiler, and can be used in code correctly. If they were quoted, then the code wouldn't be able to reference them because the compiler would change the variable names in code. If the properties to html elements were not quoted then the compiler would mangle them which would result in not-working behaviour. For example:

import { render } from 'preact'

const Component = ({ hello, world }) => {
  return <div onClick={() => {
    console.log(hello)
  }} id={world} />
}

render(<Component hello="world" world="jsx" />, document.body)

import { render } from 'preact'

const Component = ({ hello, world }) => {
  return h('div',{'onClick':() => {
    console.log(hello)
  }, 'id':world })
}

render(h(Component,{hello:"world", world:"jsx" }), document.body)

Moreover, GCC does not recognise the JSX files as source files, and the module resolution like import ExampleComponent from './example-component' does not work. Therefore, Depack will generate a temp directory with the source code where the extension is added to the files. In future, it would be easier if the compiler just allowed to pass supported recognised extensions, or added JSX to their list.

Bundle mode is perfect for creating bundles for the web, be it JSX Preact components (we only focus on Preact because our opinion is that Facebook is evil). Depack was created exactly to avoid all the corporate tool-chains etc that the internet is full of, and GCC is supported by create-react-app anyhow.

Compile Mode

The compile mode is used to create Node.JS executable binaries. This is useful when a program might have many dependencies and it is desirable to publish the package without specifying any of them in the "dependencies" field of the package.json to speed up the install time and reduce the overall linking time in the package.

Depack will recursively scan the files to detect import from and export from statements to construct the dependency list since the Google Closure Compile requires to pass all files (both source and paths to package.jsons) used in compilation as arguments. Whenever an external dependency is detected, its package.json is inspected to find out either the module or main fields. In case when the main is found, the additional --process_common_js_modules will be set.

The main problem that Depack solves is allowing to require internal Node.JS modules, e.g., import { createReadStream } from 'fs'. Traditionally, this was impossible because the compiler does not know about these modules and there is no way to pass the location of their package.json files. The strategy adopted by this software is to create proxies for internal packages in node_modules folder, for example:

// node_modules/child_process/index.js
export default child_process
export const {
  ChildProcess,
  exec,
  execFile,
  execFileSync,
  execSync,
  fork,
  spawn,
  spawnSync,
} = child_process

// node_modules/child_process/package.json
{
  "name": "child_process",
  "main": "index.js"
}

The externs for internal modules are then passed in the arguments list, allowing the compiler to know how to optimise them. Finally, the wrapper is added to prepend the output of the compiler with the actual require calls:

const path = require('path');
const child_process = require('child_process');
const vm = require('vm');
const _module = require('module'); // special case
%output%

There is another step which involves patching the dependencies which specify their main and module fields as the path to the directory rather than the file, which GCC does not understand.

Put all together, to compile the following file that contains different kinds of modules:

import { createReadStream } from 'fs' // NodeJS module
import loading from 'indicatrix' // Dependency

const load = async () => {
  const packageJson = require.resolve('depack/package.json')
  const rs = createReadStream(packageJson)
  const depack = await new Promise((r) => {
    const d = []
    rs.on('data', data => d.push(data))
    rs.on('close', () => r(d.join('')))
  })
  const { 'version': version } = JSON.parse(depack)
  return version
}
const run = async () => {
  const l = load()
  const version = await loading('Depack version is loading', l)
  console.log(version)
}
(async () => {
  await run()
})()

The next Depack command can be used:

depack example/example.js -c -V -a -w -p
# -c:      set mode to compile
# -V:      verbose output to print all flags and options
# -a:      allow for advanced compilation
# -w:      don't print warnings
# -p:      add formatting PRETTY_PRINT

# [-I 2018]: (default) set source code language to ECMA2018
# [-O 2018]: (default) set output language to ECMA2017

Modules' externs: node_modules/indicatrix/types/externs.js
java -jar /Users/zavr/node_modules/google-closure-compiler-java/compiler.jar \
--compilation_level ADVANCED --language_out ECMASCRIPT_2018 --create_source_map \
%outname%.map --formatting PRETTY_PRINT --warning_level QUIET --js_output_file \
example/generated-1.js --package_json_entry_names module,main --entry_point \
example/example.js --externs node_modules/@externs/nodejs/v8/fs.js --externs \
node_modules/@externs/nodejs/v8/stream.js --externs \
node_modules/@externs/nodejs/v8/events.js --externs \
node_modules/@externs/nodejs/v8/url.js --externs \
node_modules/@externs/nodejs/v8/global.js --externs \
node_modules/@externs/nodejs/v8/global/buffer.js --externs \
node_modules/@externs/nodejs/v8/nodejs.js --externs \
node_modules/indicatrix/types/externs.js --module_resolution NODE --output_wrapper \
#!/usr/bin/env node
'use strict';
const fs = require('fs');%output% --js \
node_modules/indicatrix/package.json node_modules/indicatrix/src/index.js \
node_modules/fs/package.json node_modules/fs/index.js example/example.js
Running Google Closure Compiler 20200112

#!/usr/bin/env node
'use strict';
const fs = require('fs');             
const h = fs.createReadStream;
async function k(a) {
  const {interval:d = 250, writable:e = process.stdout} = {};
  a = "function" == typeof a ? a() : a;
  const b = e.write.bind(e);
  var c = process.env.INDICATRIX_PLACEHOLDER;
  if (c && "0" != c) {
    return b("Depack version is loading<INDICATRIX_PLACEHOLDER>"), await a;
  }
  let f = 1, g = `${"Depack version is loading"}${".".repeat(f)}`;
  b(g);
  c = setInterval(() => {
    f = (f + 1) % 4;
    g = `${"Depack version is loading"}${".".repeat(f)}`;
    b(`\r${" ".repeat(28)}\r`);
    b(g);
  }, d);
  try {
    return await a;
  } finally {
    clearInterval(c), b(`\r${" ".repeat(28)}\r`);
  }
}
;const l = async() => {
  var a = require.resolve("depack/package.json");
  const d = h(a);
  a = await new Promise(e => {
    const b = [];
    d.on("data", c => b.push(c));
    d.on("close", () => e(b.join("")));
  });
  ({version:a} = JSON.parse(a));
  return a;
}, m = async() => {
  var a = l();
  a = await k(a);
  console.log(a);
};
(async() => {
  await m();
})();


//# sourceMappingURL=generated-1.js.map

Usage

There are Depack specific flags that can be passed when compiling a Node.JS executable. These are:

Argument	Short	Description
--compile	-c	Set the Depack mode to "compile" to create a Node.JS binary. Adds the #!usr/bin/env node at the top and sets +x permission.
--no-strict	-s	Whether to remove the "use strict" from the output.

Troubleshooting

There are going to be times when the program generated with GCC does not work. The most common error that one would get is going to be similar to the following one:

TypeError: Cannot read property 'join' of undefined
    at Ub (/Users/zavr/depack/depack/compile/depack.js:776:25)
    at Zb (/Users/zavr/depack/depack/compile/depack.js:816:13)
    at <anonymous>

This means the compiler has mangled some property on either the built-in Node.JS or external module that broke the contract with the API. This could have happened due to the incorrect/out-of-date externs that are used in Depack. In our case, we tried to access the spawnargs property on the ChildProcess in the spawncommand package, but it was undocumented, therefore the externs did not contain a record of it.

const proc = spawn(command, args, options)
proc.spawnCommand = proc.spawnargs.join(' ')

The compiler will typically produce a warning when it does not know about referenced properties which is an indicator that you might end up with runtime errors:

node_modules/@depack/depack/node_modules/spawncommand/src/index.js:54:
WARNING - Property spawnargs never defined on _spawncommand.ChildProcessWithPromise
  proc.spawnCommand = proc.spawnargs.join(' ')
                           ^^^^^^^^^

It might be difficult to understand where the problem is coming from when the source is obfuscated, especially when using external packages that the developer is not familiar with. To uncover where the problem really happens, one needs to compile the file without the source map and with pretty-print formatting using the -S -p options, and setup the debug launch configuration to stop at the point where the error happens:

{
  "type": "node",
  "request": "launch",
  "name": "Launch Transform",
  "program": "${workspaceFolder}/output/transform.js",
  "console": "integratedTerminal",
  "skipFiles": [
    "<node_internals>/**/*.js"
  ]
},

When the program is stopped there, it is required to hover over the parent of the object property that does not exist and see what class it belongs to. Once it's been identified, the source of the error should be understood which leads to the last step of updating the externs.

Compiling without source maps will show how the property was mangled, however adding the source maps will point to the location of the problem precisely. However, in this particular case the source maps didn't even work for us.

We've found out that spawnargs was mangled because it was not defined in the externs files. There can be two reasons:

• firstly, incomplete externs. The solution in the first case is to fork and patch Depack/externs and link them in your project. It is also possible to can create a separate externs file, where the API is extended, e.g.,

  // externs.js
  /** @type {!Array<string>} */
  child_process.ChildProcess.prototype.spawnargs;
  

  The program can then be compiled again by pointing to the externs file with the --externs flag:

  depack source.js -c -a --externs externs.js
  

• secondly, using undocumented APIs. Fixed by not using these APIs, or to access the properties using the bracket notation such as proc['spawnargs']. However in this case, the @suppress annotation must be added

  // Suppresses the warnings
  // spawncommand/src/index.js:54: WARNING - Cannot do '[]' access on a struct
  // proc.spawnCommand = proc['spawnargs'].join(' ')
  //                          ^^^^^^^^^^^
  /** @suppress {checkTypes} */
  proc.spawnCommand = proc['spawnargs'].join(' ')
  return proc
  

Bugs In GCC

In might be the case that externs are fine, but the Google Closure Compiler has a bug in it which leads to incorrect optimisation and breaking of the program. These cases are probably rare, but might happen. If this is so, you need to compile without -a (ADVANCED optimisation) flag, which will mean that the output is very large. Then you can try to investigate what went wrong with the compiler by narrowing down on the area where the error happens and trying to replicate it in a separate file, and using -d debug.txt Depack option when compiling that file to save the output of each pass to the debug.txt file, then pasting the code from each step in there to Node.JS REPL and seeing if it outputs correct results.

External APIs

When reading and writing files from the filesystem such as a package.json files, or loading JSON data from the 3rd party APIs, their properties must be referred to using the quoted notation, e.g.,

// reading
const content = await read(packageJson)
const {
  'module': mod,
  'version': version,
} = JSON.parse(f)

// writing
await write('package.json', {
  'module': 'test/index.mjs',
})

// loading API
const { 'results': results } = await request('https://service.co/api')

because otherwise the properties' names get changed by the compiler and the result will not be what you expected it to be. In case of loading external APIs, it's a good idea to create an extern file and defining the known properties there:

Externs	Source
// externs/api.js /** @const */ var _externalAPI /** @type {!Array<string>} */ _externalAPI.results	// source.js const { results } = /** @type {_externalAPI} */ ( await request('https://service.co/api') ) // cast the type by surrounding it with ( )

API

This package only publishes a binary. The API is available via the @Depack/depack package.

import { Bundle, Compile } from '@depack/depack'

(async () => {
  await Bundle(...)
  await Compile(...)
})

Wiki

Our Wiki contains the following pages:

• 🗼Babel Modules: Talks about importing Babel-compiled modules from ES6 code.
• 🎭CommonJS Compatibility: Discusses how to import CommonJS modules from ES6 code.
• 🐞Bugs: Lists some of the minor known bugs in the compiler.

Org Structure

• [dpck] The GitHub org.
• [@depack] The NPM scope.
• [@depack/form] Bootstrap Form.
• [@depack/router] Front-end router.
• [@depack/render] Render server-side HTML from JSX.
• [@depack/context] Testing context for unit-testing.

Notes

• The static analysis might discover built-in and other modules even if they were not used, since no tree-shaking is performed.
• [2 March 2019] Current bug does not let compile later jsx detection. Trying to compile front-end bundler with Depack.

Copyright & License

GNU Affero General Public License v3.0

	© Art Deco™ for Depack 2020