builder - npm Package Compare versions

Comparing version 2.9.1 to 2.10.0

HISTORY.md

 History
 =======
 
+## 2.10.0
+
+* Add `--unlimited-buffer` shell output option.
+
 ## 2.9.1
@@ -5,0 +9,0 @@

lib/args.js

@@ -74,2 +74,7 @@ "use strict";
       default: "info"
+    },
+    "unlimited-buffer": {
+      desc: "Use an unlimited buffer for shell commands",
+      types: [Boolean],
+      default: false
     }
@@ -76,0 +81,0 @@ },

lib/runner.js

@@ -160,5 +160,7 @@ "use strict";
 var run = function (cmd, shOpts, opts, callback) {
+  var maxBuffer = opts.unlimitedBuffer ? Infinity : MAX_BUFFER;
+
   // Update shell options and ensure basic structure.
   shOpts = _.extend({
-    maxBuffer: MAX_BUFFER,
+    maxBuffer: maxBuffer,
     env: {}
@@ -165,0 +167,0 @@ }, shOpts);

package.json

 {
   "name": "builder",
-  "version": "2.9.1",
+  "version": "2.10.0",
   "description": "An NPM-based task runner",
@@ -5,0 +5,0 @@ "repository": {

README.md

@@ -54,5 +54,5 @@ [![Travis Status][trav_img]][trav_site]
 
-* `NODE_PATH`, `PATH` enhancements to run, build, import from archetypes so
-  task dependencies and configurations don't have to be installed directly in a
-  root project.
+* `NODE_PATH`, `PATH` enhancements and module patterns to run, build, import
+  from archetypes so task dependencies and configurations don't have to be
+  installed directly in a root project.
 * A task runner capable of single tasks (`run`) or multiple concurrent tasks
@@ -100,3 +100,3 @@ (`concurrent`).
 
-```
+```sh
 $ /GLOBAL/PATH/TO/builder
@@ -228,2 +228,3 @@ [builder:local-detect] Switched to local builder at: ./node_modules/builder/bin/builder-core.js
 * `--log-level`: Level to log at (`info`, `warn`, `error`, `none`)
+* `--unlimited-buffer`: Unlimited shell output buffer.
 * `--expand-archetype`: Expand `node_modules/<archetype>` with full path (default: `false`)
@@ -252,2 +253,3 @@ * `--builderrc`: Path to builder config file (default: `.builderrc`)
 * `--log-level`: Level to log at (`info`, `warn`, `error`, `none`)
+* `--unlimited-buffer`: Unlimited shell output buffer.
 * `--expand-archetype`: Expand `node_modules/<archetype>` with full path (default: `false`)
@@ -295,2 +297,3 @@ * `--builderrc`: Path to builder config file (default: `.builderrc`)
 * `--log-level`: Level to log at (`info`, `warn`, `error`, `none`)
+* `--unlimited-buffer`: Unlimited shell output buffer.
 * `--expand-archetype`: Expand `node_modules/<archetype>` with full path (default: `false`)
@@ -753,3 +756,3 @@ * `--builderrc`: Path to builder config file (default: `.builderrc`)
 
-#### Initializing your project
+#### Initializing a Project
 
@@ -826,11 +829,105 @@ An archetype is simply a standard npm module with a valid `package.json`. To set
 
-#### NOTE: Application vs. Archetype Dependencies
+#### Node Require Resolution and Module Pattern
 
-While we would love to have `builder` manage _all_ the dependencies of an
-application, the practical realities of how npm works is that archetypes can
-only manage dependencies for `scripts` commands **run by a `builder` command**.
-`builder` mutates `PATH` and `NODE_PATH` to include archetype dependencies, but
-without this `builder` magic, ordinary code won't otherwise be able to use
-archetype dependencies.
+As a background primer, whenever a file has a `require("lib-name")` in it, Node
+performs the following check for `/path/to/ultimate/file.js`:
 
+```
+/path/to/ultimate/node_modules/lib-name
+/path/to/node_modules/lib-name
+/path/node_modules/lib-name
+/node_modules/lib-name
+```
+
+After this, Node then checks for `NODE_PATH` for additional paths to search.
+This presents a potentially awkward pattern when combined with npm
+deduplication / flattening for say a file like:
+`<root>/node_modules/<archetype>/config/my-config.js` that requires
+`lib-name@right-version` as follows:
+
+Node modules layout:
+
+```
+<root>/
+  node_modules/
+    lib-name@wrong-version
+    <archetype>/
+      config/my-config.js         // require("lib-name");
+    <archetype-dev>/
+      node_modules/
+        lib-name@right-version
+```
+
+This unfortunately means that the search path for `require("lib-name")` is:
+
+```
+# From file path priority resolution
+<root>/node_modules/<archetype>/config/node_modules
+<root>/node_modules/<archetype>/node_modules
+<root>/node_modules/node_modules
+<root>/node_modules                                   // Matches `lib-name@wrong-version`!!!
+
+# Now, from `NODE_PATH`
+<root>/node_modules/<archetype>/node_modules
+<root>/node_modules/<archetype-dev>/node_modules      // Too late for `right-version`.
+```
+
+##### The Module Pattern
+
+To remedy this situation, we encourage a very simple pattern to have Node.js
+`require`'s start from the dev archetype when appropriate by adding a one-line
+file to the dev archetype: `<archetype-dev>/require.js`
+
+```js
+// Contents of <archetype-dev>/require.js
+module.exports = require;
+```
+
+By exporting the `require` from the dev archetype, the resolution starts in the
+dev archetype and thus ensures the dev archetype "wins" for the archetype tasks.
+Thus in any archetype files that do a `require`, simply switch to:
+
+```js
+var mod = require("<archetype-dev>/require")("lib-name");             // Module
+var modPath = require("<archetype-dev>/require").resolve("lib-name"); // Module path
+```
+
+And the dependency from the dev archetype is guaranteed to "win" no matter what
+happens with actual module layout from npm installation.
+
+_Note_ that because a file from within the normal `<archetype>` will naturally
+search `<archetype>/node_modules` before hitting `<root>/node_modules` you do
+not need to use this `require` pattern for normal archetype dependencies in
+archetype Node.js files.
+
+Node.js files in the normal production archetype do not need a
+`<archetype>/require.js` file akin to the dev archetype because
+`<archetype>/node_modules` is already at the top of the require search path.
+However, some projects may wish to have an archetype control _and provide_
+application dependencies and dev dependencies, which we discuss in the
+[next section](#application-vs-archetype-dependencies)
+
+###### ES.next Imports and The Module Pattern
+
+The module pattern works great for any `require()`-based CommoneJS code.
+Unfortunately, when using babel and ES.next imports like:
+
+```js
+import _ from "lodash";
+```
+
+The module pattern is _not_ available because the actual `require("lodash")`
+statement spit out during transpilation is not directly accessible to the
+developer.
+
+We have [ticket #111](https://github.com/FormidableLabs/builder/issues/111) out
+to write a babel plugin to make the module pattern semantics available during
+babel transpilation as well.
+
+#### Application vs. Archetype Dependencies
+
+Out of the box `builder` does not manage application dependencies, instead
+managing dependencies only for builder _workflows_ and _tasks_, e.g. things
+starting with the `builder` command.
+
 Most notably, this means that if your _application_ code includes a dependency
@@ -846,3 +943,3 @@ like `lodash`:
 
-and the root project is consumed in _anything besides a `builder` command_,
+and the root project is consumed by _anything besides a `builder` command_,
 then it **must** have a dependency like:
@@ -857,9 +954,48 @@
 
-And, _even if_ your `<archetype>/package.json` also includes the exact same
-dependency.
+However, if you want to use builder to _also_ manage application dependencies,
+then you can follow [the module pattern](#the-module-pattern) and provide an
+`<archetype>/require.js` file consisting of:
 
-This rule applies to even simple scenarios such as the root project being
-published to npm, after which other users will rely on the code outside of
-`builder` processes.
+```js
+// Contents of <archetype>/require.js
+module.exports = require;
+```
 
+The root project could then require code like:
+
+
+```js
+var modFromProd = require("<archetype>/require")("lib-name");               // Module
+var pathFromProd = require("<archetype>/require").resolve("lib-name");      // Module path
+var modFromDev = require("<archetype-dev>/require")("lib-name");            // Module
+var pathFromDev = require("<archetype-dev>/require").resolve("lib-name");   // Module path
+```
+
+Using the above pattern, `<archetype>` or `<archetype-dev>` dependencies would
+override `<root>/node_modules` dependencies reliably and irrespective of npm
+flattening.
+
+So, turning back to our original example, we could utilize archetype
+dependencies by refactoring to something like:
+
+
+```js
+// <root>/src/index.js
+var _ = require("<archetype>/require")("lodash");
+
+module.exports = _.camelCase("Hi There");
+```
+
+and dev code like:
+
+
+```js
+// <root>/test/index.js
+var _ = require("<archetype-dev>/require")("lodash");
+
+module.exports = _.camelCase("Hi There");
+```
+
+after which you would _not_ need a `lodash` dependency in `root/package.json`.
+
 #### Moving `dependencies` and `scripts` to a New Archetype
@@ -884,4 +1020,7 @@
 you have moved to the archetype. However, take care to
-[not remove real application dependencies](#note-application-vs-archetype-dependencies).
+[not remove real application dependencies](#application-vs-archetype-dependencies)
+unless you are using a module pattern to provide
+[application dependencies](#application-dependencies).
 
+
 ##### Moving `scripts` and Config Files
@@ -958,3 +1097,4 @@
     files: [
-      require.resolve("sinon/pkg/sinon"),
+      require.resolve("sinon/pkg/sinon"),                             // Normal archetype
+      require("<archetype-dev>/require").resolve("sinon/pkg/sinon"),  // Dev archetype
       MAIN_PATH
@@ -991,2 +1131,3 @@ ],
 │   └── package.json
+│   └── require.js
 └── package.json
@@ -1007,10 +1148,13 @@ ```
 1. `<cwd>/node_modules/<archetype>/.bin`
-2. `<cwd>/node_modules/.bin`
-3. Existing `PATH`
+2. `<cwd>/node_modules/<archetype-dev>/.bin`
+3. `<cwd>/node_modules/.bin`
+4. Existing `PATH`
 
-`NODE_PATH`:
+`require` + `NODE_PATH`: For `file.js` with a `require`
 
-1. `<cwd>/node_modules/<archetype>/node_modules`
-2. `<cwd>/node_modules`
-3. Existing `NODE_PATH`
+1. `/PATH/TO/file.js` (all sub directories + `node_modules` going down the tree)
+2. `<cwd>/node_modules/<archetype>/node_modules`
+3. `<cwd>/node_modules/<archetype-dev>/node_modules`
+4. `<cwd>/node_modules`
+5. Existing `NODE_PATH`
 
@@ -1021,2 +1165,19 @@ The order of resolution doesn't often come up, but can sometimes be a factor
 
+### Alternative to `npm link`
+
+In some cases, `npm link` can interfere with the order of resolution. If you
+run into resolution problems, you can develop locally with the
+following in your consuming project's `package.json` as an alternative to `npm link`:
+
+```json
+{
+  "dependencies": {
+    "YOUR_ARCHETYPE_NAME": "file:../YOUR_ARCHETYPE_REPO"
+  },
+  "devDependencies": {
+    "YOUR_ARCHETYPE_NAME_dev": "file:../YOUR_ARCHETYPE_REPO/dev"
+  }
+}
+```
+
 ### Project Root
@@ -1136,2 +1297,5 @@
   directory).
+* Replace all instances of `require("<archetype-dev>/require")` and
+  `require("<archetype>/require")` with `require` in configuration / other
+  Node.js files from the archetype.
 * Review all of the combined `scripts` tasks and:
@@ -1138,0 +1302,0 @@ * resolve duplicate task names

New alerts

Shell access

Supply chain risk

This module accesses the system shell. Accessing the system shell increases the risk of executing arbitrary code.

Found 1 instance in 1 package

Dynamic require

Supply chain risk

Dynamic require can indicate the package is performing dangerous or unsafe dynamic code execution.

Found 1 instance in 1 package

Environment variable access

Supply chain risk

Package accesses environment variables, which may be a sign of credential stuffing or data theft.

Found 3 instances in 1 package

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

Long strings

Supply chain risk

Contains long string literals, which may be a sign of obfuscated or packed code.

Found 1 instance in 1 package

Fixed alerts

Shell access

Supply chain risk

This module accesses the system shell. Accessing the system shell increases the risk of executing arbitrary code.

Found 1 instance in 1 package

Dynamic require

Supply chain risk

Dynamic require can indicate the package is performing dangerous or unsafe dynamic code execution.

Found 1 instance in 1 package

Environment variable access

Supply chain risk

Package accesses environment variables, which may be a sign of credential stuffing or data theft.

Found 3 instances in 1 package

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

Long strings

Supply chain risk

Contains long string literals, which may be a sign of obfuscated or packed code.

Found 1 instance in 1 package

Improved metrics

Total package byte prevSize

96019

6.49%

Lines of code

1468

0.41%

Number of lines in readme file

1316

14.24%

No dependency changes