thread-loader

What is thread-loader?

The thread-loader package is used to offload expensive loaders to a worker pool. It can be particularly useful when dealing with resource-intensive loaders in a webpack build process, as it can help to speed up compilation by parallelizing the work.

What are thread-loader's main functionalities?

Offloading Loaders to Worker Pool

This feature allows you to offload loaders like 'babel-loader' to a worker pool. The code sample shows how to use 'thread-loader' in a webpack configuration file to process JavaScript files with 'babel-loader' in a separate thread.

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        include: path.resolve('src'),
        use: [
          'thread-loader',
          'babel-loader'
        ]
      }
    ]
  }
};

Custom Worker Pool

This feature allows you to specify the number of workers in the pool. The code sample demonstrates how to set up a custom worker pool with two workers using 'thread-loader' options.

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        include: path.resolve('src'),
        use: [
          {
            loader: 'thread-loader',
            options: {
              workers: 2
            }
          },
          'babel-loader'
        ]
      }
    ]
  }
};

Worker Pool Warm-up

This feature allows you to warm up the worker pool before running the actual loaders. The code sample shows how to use 'thread-loader' to pre-load 'babel-loader' and '@babel/preset-env' to the worker pool.

const threadLoader = require('thread-loader');

const warmupOptions = {
  // pool options, like passed to loader options
  // must match loader options to boot the correct pool
};

threadLoader.warmup(warmupOptions, [
  // modules to load
  // can be any module, i. e.
  'babel-loader',
  '@babel/preset-env'
]);

Other packages similar to thread-loader

happypack

HappyPack is a package similar to thread-loader that also enables parallel processing of files in webpack. It works by transforming the files in parallel using worker threads and then compiling them together. Compared to thread-loader, HappyPack provides more detailed configuration options but is no longer actively maintained.

parallel-webpack

Parallel-webpack allows you to run multiple instances of webpack in parallel, which can significantly speed up the build process. It differs from thread-loader in that it parallelizes the entire build process rather than individual loaders. This can be more efficient for large projects with many entry points.

cache-loader

Cache-loader is a webpack loader that caches the result of expensive loader functions on disk. While it doesn't parallelize work like thread-loader, it can speed up subsequent builds by reusing previous results, thus reducing the need to run the loaders again.

README

thread-loader

Runs the following loaders in a worker pool.

Install

npm install --save-dev thread-loader

Usage

Put this loader in front of other loaders. The following loaders run in a worker pool.

Loaders running in a worker pool are limited. Examples:

• Loaders cannot emit files.
• Loaders cannot use custom loader API (i. e. by plugins).
• Loaders cannot access the webpack options.

Each worker is a separate node.js process, which has an overhead of ~600ms. There is also an overhead of inter-process communication.

Use this loader only for expensive operations!

Examples

webpack.config.js

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        include: path.resolve("src"),
        use: [
          "thread-loader",
          // your expensive loader (e.g babel-loader)
        ]
      }
    ]
  }
}

with options

use: [
  {
    loader: "thread-loader",
    // loaders with equal options will share worker pools
    options: {
      // the number of spawned workers, defaults to (number of cpus - 1) or
      // fallback to 1 when require('os').cpus() is undefined
      workers: 2,

      // number of jobs a worker processes in parallel
      // defaults to 20
      workerParallelJobs: 50,

      // additional node.js arguments
      workerNodeArgs: ['--max-old-space-size=1024'],

      // Allow to respawn a dead worker pool
      // respawning slows down the entire compilation
      // and should be set to false for development
      poolRespawn: false,

      // timeout for killing the worker processes when idle
      // defaults to 500 (ms)
      // can be set to Infinity for watching builds to keep workers alive
      poolTimeout: 2000,

      // number of jobs the poll distributes to the workers
      // defaults to 200
      // decrease of less efficient but more fair distribution
      poolParallelJobs: 50,

      // name of the pool
      // can be used to create different pools with elsewise identical options
      name: "my-pool"
    }
  },
  // your expensive loader (e.g babel-loader)
]

prewarming

To prevent the high delay when booting workers it possible to warmup the worker pool.

This boots the max number of workers in the pool and loads specified modules into the node.js module cache.

const threadLoader = require('thread-loader');

threadLoader.warmup({
  // pool options, like passed to loader options
  // must match loader options to boot the correct pool
}, [
  // modules to load
  // can be any module, i. e.
  'babel-loader',
  'babel-preset-es2015',
  'sass-loader',
]);

Maintainers

sokra

Changelog

2.1.0 (2018-12-21)

Features

• add poolRespawn flag to speed up incremental builds (#52) (76535bf)

<a name="2.0.2"></a>