@code-engine/workers

CodeEngine workers

This library is used inside CodeEngine to provide multi-threaded concurrency. It exports a WorkerPool class, which manages worker threads and uses them to process files.

NOTE: This is an internal library that is only intended to be used by CodeEngine. Using it outside of CodeEngine is discouraged.

WorkerPool class

This class creates worker threads, manages their lifecycle, and transfers CodeEngine files back and forth between threads for processing.

import WorkerPool from "@code-engine/workers";

// Create a new WorkerPool instance with 4 workers
let pool = new WorkerPool(4, context);

try {
  // Import a FileProcessor plugin in all 4 workers
  let processFile = await pool.importFileProcessor("./my-file-processor.js");

  // Process a file on one of the workers
  await processFile(myFile, context);
}
finally {
  // Safely dispose the pool and threads
  await pool.dispose();
}

WorkerPool constructor

• concurrency: The number of worker threads to create

• context: A CodeEngine Context object.

import WorkerPool from "@code-engine/workers";

// Create a new WorkerPool instance with 4 workers
let pool = new WorkerPool(4, context);

WorkerPool.size

Read-only property that returns the number of worker threads in the pool. After the dispose() method is called, this property will always return zero.

import WorkerPool from "@code-engine/workers";

let pool = new WorkerPool(4, context);
console.log(pool.size);   // 4

await pool.dispose();
console.log(pool.size);   // 0

WorkerPool.isDisposed

Indicates whether the dispose() method has been called. Once disposed, the WorkerPool instance is no longer usable.

import WorkerPool from "@code-engine/workers";

let pool = new WorkerPool(4, context);
console.log(engine.isDisposed);     // false

await engine.dispose();
console.log(engine.isDisposed);     // true

WorkerPool.importFileProcessor(module)

Imports a CodeEngine FileProcessor plugin in all worker threads.

• module: A module name, path, or ModuleDefinition object. The module must export a FileProcessor function.

import WorkerPool from "@code-engine/workers";
let pool = new WorkerPool(4, context);

// Import a FileProcessor plugin in all 4 workers
let processFile = await pool.importFileProcessor("./my-file-processor.js");

// Process a file on one of the workers
await processFile(myFile, context);

WorkerPool.importModule(module)

Imports a JavaScript module in all worker threads. The module export (if any) is ignored. This method is intended for loading polyfills, globals, hooks, and other modules with side-effects.

• module: A module name, path, or ModuleDefinition object.

import WorkerPool from "@code-engine/workers";
let pool = new WorkerPool(4, context);

// Import a polyfill module in all worker threads
await pool.importModule("@babel/polyfill");

WorkerPool.dispose()

Terminates the worker threads and releases all system resources that are held by a WorkerPool instance. Once dispose() is called, the WorkerPool instance is no longer usable.

import WorkerPool from "@code-engine/workers";

let pool = new WorkerPool(4, context);
await pool.dispose();

"error" event

This event is fired whenever an unhandled error occurs in any of the worker threads. If you don't handle this event, then Node.js will automatically terminate the process.

NOTE: When an unhandled error occurs, the WorkerPool instance and/or its worker threads may be left in an invalid or unusable state. For this reason, we recommend that you dispose the WorkerPool instance and stop using it.

import WorkerPool from "@code-engine/workers";
let pool = new WorkerPool();

pool.on("error", (error) => {
  console.error("An unhandled error occurred:", error);
  pool.dispose();
});

Contributing

Contributions, enhancements, and bug-fixes are welcome! File an issue on GitHub and submit a pull request.

Building

To build the project locally on your computer:

• Clone this repo
  git clone https://github.com/CodeEngineOrg/code-engine-workers.git

• Install dependencies
  npm install

• Build the code
  npm run build

• Run the tests
  npm test

License

@code-engine/workers is 100% free and open-source, under the MIT license. Use it however you want.

Big Thanks To

Thanks to these awesome companies for their support of Open Source developers ❤