polestar

Polestar

A commonjs-ish module loader for browsers, as used in Demoboard.

Polestar loads commonjs modules from NPM and/or a virtual file system on-demand. It is highly configurable, allowing you to:

• Resolve and build each require() call's request asynchronously (while your modules still use require() as a synchronous function)
• Set appropriate global variables for your modules, including process, global, or a stubbed window object.
• Capture errors while loading modules, and forward them as appropriate.
• Display a loading indicator until the entry point is ready to execute.

yarn add polestar

Usage

import { Polestar } from 'polestar'

let polestar = new Polestar({
  /**
   * Any keys set here will be available as globals within all executed code.
   * 
   * You can use this to provide process, global, setImmediate, etc., or to
   * provide stubs for window, history, etc.
   */
  globals: {
    process: {
      env: {},
    }
  },

  /**
   * Sets the value of `this` within loaded module code
   */
  moduleThis: window,

  /**
   * Fetches required modules. See the "Fetchers" section below for details
   * on how this function should behave.
   */
  fetcher: async (url: string, meta: FetcherMeta) =>
    api.fetchModule(url, meta),

  /**
   * The resolver is responsible for taking a string passed to `require`, and
   * turning it into an id of an already loaded module, or a URL that will be
   * passed to the fetcher.
   * 
   * The default resolver can handle `npm://` URLs, standard HTTP urls,
   * "vfs://" URLs and webpack-style loader strings, so you can probably omit
   * this option.
   */
  resolver: defaultResolver,

  /**
   * Called once the first required module is ready to execute, but before it
   * has actually executed. Useful for deciding when to show/hide loading
   * indicators.
   */
  onEntry: () => {
    api.dispatch('entry')
  },

  /**
   * Called when for each error that occurs while executing a module, while
   * fetching a module, etc.
   */
  onError: (error) => {
    console.error(error)
  },
})

/**
 * `polestar.require` will fetch a module's dependencies, then execute the
 * module, before finally returning a promise to the module object.
 */
let indexModule = await polestar.require('vfs:///index.js')

/**
 * You can require any URL that your fetch function supports.
 */
polestar.require('npm:///some-package@latest')
polestar.require('vfs:///index.js')
polestar.require('some-loader!vfs:///index.css')

/**
 * `evaluate` takes a string and a list of dependencies. Once the dependencies
 * are available, it'll execute the string as a module, and return the module
 * object.
 */
let anotherModule = await polestar.evaluate(
  ['react', 'react-dom'],
  `
  var React = require('react');
  var ReactDOM = require('react-dom');
  ReactDOM.render(
    React.createElement('div', {}, "Hello, world!"),
    document.getElementById('root')
  )
  `
)

Requests, URLs & Module ids

Polestar uses three different types of strings to reference modules.

Requests

Requests are the strings that appear within require() statements. They can take any number of formats:

• Paths relative to the module that contains the require() call, e.g. ./App.js
• Fully qualified URLs, e.g. https://unpkg.com/react
• Bare imports, e.g. react
• They can be prefixed have webpack-style loader, e.g. style-loader!css-loader!./styles.css

Whenever Polestar encounters a request, it first resolves it to either a URL, or a module id.

URLs

In polestar, a URL is a string generated by the resolver that can be passed to the fetcher to request a module's source and dependencies.

The default resolver is able to create two types of URLs:

• NPM URLs, e.g. npm://react@latest, are generated from bare imports. They always contain a package name and version range string (e.g. @latest, @^1.7.2, @16.7.0). They can also optionally contain a path.
• Other URLs are treated as relative to the module making the request.

One issue with using URLs to refer to modules is that URLs can redirect to other URLs, so a single module can be referenced by multiple different URLs. For example, each of these unpkg URLs may refer to the same module:

• https://unpkg.com/react
• https://unpkg.com/react@16.6.3
• https://unpkg.com/react@latest
• https://unpkg.com/react@16.6.3/index.js

Because of this, polestar doesn't index modules by URL. Instead, it indexes modules by ID.

Module ids

When the fetcher returns a result for a given URL, it'll also include an ID. This ID must be a URL, and is usually the URL at the end of any redirect chain that the fetcher encounters.

Once Polestar has loaded the module, it'll notify the resolver of the ID of the new module, as well as any URLs that map to that ID. This allows the resolver to map requests to already-loaded modules where possible, speeding up load time for frequently referenced modules like react.

A modules relative requires (e.g. require('./App.js)) will also be resolved relative to the module ID.

Fetchers

A minimal fetcher function just takes a URL and returns the url, id, dependencies and source of the the module at that URL.

const fetcher = (url: string) => ({
  url: 'vfs:///Hello.js',

  // The canonical URL of this source file, 
  id: 'vfs:///Hello.js',

  // An array of requests that can be made by `require()` within this module.
  // You can find these using a package like babel-plugin-detective.
  dependencies: ['react']

  // The source that will be evaluated for the module.
  code: `
    var React = require('react')

    module.exports.Hello = function() {
      return React.createElement('h1', {}, 'hello')
    }
  `,
})

Fetcher functions also receive a meta object with information on where the fetch originated, which is useful for error messages.

type Fetcher = (url: string, meta: FetchMeta) => Promise<FetchResult>

interface FetchMeta {
  requiredById: string,
  originalRequest: string
}

UMD modules

For UMD modules, the dependencies can be omitted and replaced with the string umd.

const fetcher = (url: string) => ({
  url: 'https://unpkg.com/react@latest'

  // The canonical URL of this source file. Note that you'll have to decide
  // how to match URLs to UMD ids within your fetcher function.
  id: 'https://unpkg.com/react@16.6.3/umd/react.development.js',

  // An array of requests that can be made by `require()` within this module.
  // You can find these using a package like babel-plugin-detective.
  dependencies: 'umd',

  code: `...`,
})

This works as dependencies are already specified within the UMD module. This is especially useful for large modules like react-dom, as parsing the received code to find the requires can be quite slow.

Version Ranges

Many packages require specific versions of their dependencies to work; they won't work at all if you default to using the latest versions of all packages involved.

In order to resolve bare requests to the correct version of an NPM package, the resolver needs to have access to the dependencies object from a module's package.json, for modules that define one. These dependencies should be returned via the dependencyVersionRanges property of the fetcher's result.

const fetcher = (url: string) => ({
  url: 'https://unpkg.com/react-dom@latest'
  id: 'https://unpkg.com/react-dom@16.6.3/umd/react-dom.development.js',
  code: `...`,
  dependencies: 'umd',
  dependencyVersionRanges: {
    // As read from https://unpkg.com/react-dom@16.6.3/package.json
    "loose-envify": "^1.1.0",
    "object-assign": "^4.1.1",
    "prop-types": "^15.6.2",
    "scheduler": "^0.11.2"
  }
})

Full Fetcher types

type VersionRanges = { [name: string]: string }

type Fetcher = (url: string, meta: FetchMeta) => Promise<FetchResult>

interface FetchMeta {
  requiredById: string,
  originalRequest: string
}

interface FetchResult {
  id: string,
  url: string,

  // Should already include any source map / source url
  code: string, 

  // Things that can be required by the module (as specified in require() statements)
  // If the string 'umd' is specified, the module will be treated as a umd module,
  // and it's dependencies will be loaded.
  // If undefined, a `require` function will not be made available.
  dependencies?: 'umd' | string[], 

  // Hints for what versions required dependencies should resolve to
  dependencyVersionRanges?: VersionRanges,
}